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We address the role played by orbital degeneracy in strongly correlated transition metal com- 
pounds. Specifically, we study the effective spin-orbital model derived for the d^ ions in a three- 
dimensional perovskite lattice, as in KCuFs, where at each site the doubly degenerate eg orbitals 
contain a single hole. The model describes the superexchange interactions that depend on the 
pattern of orbitals occupied and shows a nontrivial couphng between spin and orbital variables at 
nearest neighbor sites. We present the ground state properties of this model, depending on the 
splitting between the Cg orbitals Ez, and the Hund's rule coupling in the excited states, Jh- The 
classical phase diagram consists of six magnetic phases which all have different orbital ordering: two 
antiferromagnetic (AF) phases with G-AF order and either — or — r^ orbitals occupied, two 
phases with mixed orbital (MO) patterns and A-AF order, and two other MO phases with either C- 
AF or G-AF order. All of them become degenerate at the multicritical point M = {E^, Jh) = (0, 0). 
Using a generalization of linear spin-wave theory we study both the transverse excitations which 
are spin-waves and spin-and-orbital-waves, as well as the longitudinal (orbital) excitations. The 
transverse modes couple to each other, and the spin-and-orbital-wave turns into a soft mode near 
the M point. Therefore, quantum corrections to the long-range-order parameter are drastically in- 
creased near the orbital degeneracy, and classical order is suppressed in a crossover regime between 
the G-AF and A-AF phases in the (i?z, Jh) plane. This behavior is reminiscent of that found in 
frustrated spin models, and we conclude that orbital degeneracy provides a new and physically real- 
izable mechanism which stabilizes a spin liquid ground state due to inherent frustration of magnetic 
interactions. We also point out that such a disordered magnetic phase is likely to be realized at low 
Jh and low electron-phonon coupling, as in LiNi02. 



PACS numbers: 71.27.-Ha, 75.10.-b, 75.30.Kz, 75.30.Ds. 



NEW MECHANISM OF FRUSTRATION 
NEAR ORBITAL DEGENERACY 



Quite generally, strongly correlated electron systems 
involve orbitally degenerate statesjll such as 3d (Ad) states 
in transition metal compounds, and 4/ (5/) states in 
rare-earth compounds. Yet, the orbital degrees of free- 
dom are ignored in most situations and the common ap- 
proach is to consider a single correlated orbital per atom 
which leads to spin degeneracy alone. Indeed, most of 
the current studies of strongly correlated electrons deal 
with models of nondegenerate orbitals. The problems 
discussed recently include mechanisms of ferromagnetism 
in the Hubbard modelBI hole propagation and quasiparti- 
cles in the t— J model,0 and magnetic states of the Kondo 
lattice.u Of course, in many actually existing compounds 
the orbital degeneracy is removed by the crystal field, and 
a single-orbital approach is valid per se. Also, from a fun- 
damental point of view it is often possible to argue that 



orbital degeneracy is qualitatively irrelevant, and that a 
single-orbital approach can capture the generic mecha- 
nisms operative in the presence of strong correlations. 

However, neither of these arguments applies for a class 
of insulating strongly correlated transition metal com- 
pounds, where the crystal field leaves the 3d orbitals ex- 
plicitly degenerate and thus the type of occupied orbitals 
is not known a priori, while the magnetic interaction be- 
tween the spins of neighboring transition metal ions de- 
pends on which orbitals are occupied. In this particular 
class of Mott-Hubbard insulators (MHI) the orbital de- 
grees of freedom acquire a separate existence in much the 
same way as the spins do. Thereby, the degeneracy of t2g 
orbitals is of less importance, as the magnetic superex- 
change and the coupling to the lattice are rather weak. 
A more interesting situation occurs when eg orbitals are 
partly occupied, which results in stronger magnetic in- 
teractions, and strong Jahn- Teller (JT) effect. Typical 
examples of such ions are: Cu^+ {d^ configuration, one 
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hole in Cg-orbitals), low-spin Ni^+ {dJ configuration, one 
electron in eg-orbitals), as well as Mn'^"'" and Cr^+ ions 
(high-spin c?^ configuration, one Cg electron). The sim- 
plest model, relevant for transition metal ions, which is 
also the subject of the present paper, was introduced by 
Kugel and Khomskii more than two decades ago,B but 
its meanjfield (MF) phase diagram was analyzed only 
recentlyjj It describes magnetic superexchange interac- 
tions between spins S = 1/2, and the accompanying or- 
bital superexchange interactions. 

One might argue that the (classical) orbital degener- 
acy is not easy to realize in such systems, as the electron- 
phonon coupling will lead to the conventional collective 
JT instability. In fact, it can be shown that the JT in- 
stability is enhanced by the orbital pattern once this, 
been established as the result of effective interactions:! 
the lattice has to react to the symmetry lowering in the 
orbital sector, which can only increase the stability of a 
given magnetic state. So the lattice follows rather than 
induces the orbital order, and therefore, as waa.|i)ointed 
out in the early work by Kugel and Khomskii,ErLl in the 
orbitally degenerate MHI one has to consider in first in- 
stance the purely electronic problem. This is supported 
by the results of recent band structure calculations us- 
ing the local density approximation (LDA) with the elec- 
tron interactions treated in Hartree-Fock approximation, 
the so-called LDA-I-U method, which permits both or- 
bitals and spins to polarize while keeping the accurate 
treatment of the electron-lattice coupling of LDA intact. 
These calculations reproduce the observed orbital order- 
ing in KCuFa (Ref. |l^ and in LaMnOg (Ref. 0), even 
when the lattice distortions are suppressed, while allow- 
ing the lattice to relax only yields an energy gain which 
is minute in comparison with the energies involved in the 
orbital ordering. 

Effects of orbital degeneracy are expected as soon as 
crystal-field splittings become small. Such situations 
are frequently encountered in rare-earth systems, where 
they lead to the aa-called singlet-triplet models discussed 
in the seventiesji3 while in the 3d oxides only a_sniall 
number of so-called Kugel-Khomskii (KK) systemaj have 
been recognized that actually exhibit orbital effects. El 
As pointed out by Kugel and Khomskii, El in such situ- 
ations the superexchange interactions have a more com- 
plex form than in spin-only models and one expects that 
also in some other Mott-Hubbard (or charge-transfer) 
insulators new magnetic phases might arise due to the 
competition of various magnetic and orbital interactions. 
Some examples of such a competition of magnetic inteiu 
actions are encountered in the heavy fermion systemspu 
and in the manganites where the phase diagrams show a 
particular frustration of magnetic interactions.Lj 113 

Even more interesting behavior is expected for the 
doped systems, as the competition between the mag- 
netic, orbital, and kinetic energy is then described by 
t — J Hamiltonians of a novel type, which exhibit qual- 
itatively different excitation spectra due to the underly- 
ing orbital degeneracyJia A few examples of such mod- 



els have already been discussed in the literature, such 
as the triplet t — J jModelJlj the low-spin defects in a 
5 = 1 backffround,E3 or a new t — J model for the 
manganitesO Whether such models are realistic enough 



1 for example inJie manganites there 
and theoretical^ indications that the 



is not yet clear, 2 
are experimentalE 
double-exchange model which includes only the spin de- 
grees of freedom is insufficient to understand the, trans, 
port properties under doping. Recent workliSliUcJa 
strongly suggests that an extension of the t — J and 
double-exchange models which include fully the orbital 
physics should be studied instead. 

In this paper we shall consider only the insulating sit- 
uation, where one can integrate out the d — d excitations 
and derive an effective low-energy Hamiltonian. This 
approach is justified by the large on-site Coulomb in- 
teraction U, being the largest energy scale in MHI. A 
low-energy Hilbert space splits off, spanned by spin and 
orbital configuration space, with superexchangelike cou- 
plings between both spin and orbital local degrees of 
freedom. The orbital sector carries a discrete symme- 
try and the net outcome is that the clock-like orbital 
degrees of freedom get coupled into the SU{2) spin prob- 
lem. The resulting low-energy Hamiltonian is called a 
spin- orbital model. Here we focus on the simplest situ- 
ation with two nearly degenerate partially filled Cg or- 
bitals, and completely filled i2a-iOrbitals, as encountered 
in KCuFg and related systemsfl These are JT-distorted 
cubic crystals, three-dimensional (3D) analogues of the 
cuprate superconductors Ej In the high-Tc cuprates, or- 
bital degeneracy would occur if the Cu-0 bonds which 
involve apical oxygens were squeezed such as to recover 
the cubic symmetry of the perovskite lattice. Of course, 
such a degeneracy of Cg orbitals is far from being real- 
ized in the aetaal high-Tc materials, and in their parent 
compounds .cZHHa 

If only one correlated orbital is present, the system 
may be described by the effective single-band Hubbard 
model (typically with niO|Ca extended hopping) , as in the 
cuprate superconductors Ej In this simplest case the ef- 
fective model at half-filling is the Heisenberg model with 
antiferromagnetic (AF) superexchange.E3 This changes 
when more than one 3d orbital is partly occupied. For 
example, we show in Sec. Q that virtual excitations in- 
volving d^ local triplet states become possible in the case 
of degenerate Cg orbitals, and this leads to additional fer- 
romagnetic (FM) interactions. The origin of these new 
interactions was first discussed, by Kugel and Khomskiitl 
and by Cyrot and Lyon-CaeiH who pointed out that the 
strongest superexchange constant results from the exci- 
tation to the lowest energy triplet state in the degenerate 
Hubbard model. 

The model proposed by Kugel and Khomskii explains 
qualitatively the observed magnetic ordering in KCuFa 
as being due to an orbital ordering which gives planes of 
perpendicularly oriented orbitals, and the magnetic cou- 
pling becomes Ifen FM according to the Goodenough- 
Kanamori rules.E2l As mentioned above, such a state was 
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indeed foundJjji the band structure calculations of Liecht- 
enstein et ali3 using the LDA+U method. An analogous 
orbital order is responsibleJor ferromagnetism in the pla- 
nar FM insulator K2CUF4.EI In the colossal magnetoresis- 
tance parent compound LaMnOs, where the Cg orbitals 
contain one electrpm instead of one hole, a similar orbital 
ordering occurs ,0113 although the situation there is more 
complex due to the presence of t2g spins, so that the re- 
sulting superexchange is npl between spins S = 1/2 but 
between total spins 5 = 20 Another example of degen- 
erate orbitals is found in V2O3, with the orbital ordering 
studied by-Castellani, Natoli and Ranningcr in a series 
of papers E3 In fact, their prediction that the transition 
into the AF insulator is accompanied by the onset of ofcj 
bital ordering was experimentally verified only recently.Efl 
However, this case is still open, as recent electronic struc- 
ture calculations suggest that doubly degenerate orbitals 
are occupied by two electrons in the high-spin state and 
the orbital degree of freedom plays no role.E3 

In any of the above situations the orbital ordering 
breaks the translational symmetry and represents an 
analogon of spin antiferromagnetism in orbital space. So, 
classically orbital ordering is expected to occur quite 
generally whenever one encounters Cg orbitals contain- 
ing either one hole or one electron, with important con- 
sequences for the magnetism. This immediately raises a 
number of questions about what happens in the quantum 
regime. Will orbital long-range order (LRO) be robust or 
will it give way to an orbital liquid, as proposed by Ishi- 
hara, Yamanaka and Nagaosa (Ref. |3^)? In either case, 
what are the consequences of the enlarged phase space 
and the associated additional channels for quantum fluc- 
tuations for the magnetism: can magnetic LRO survive 
or will it be replaced by a spin liquid! 

Quantum disordered phases are of great current in- 
terest. Spin disorder is well known to occur in one- 
dimensional (ID) and quasi ID quantum spin systems, 
and the best example is the ID Heisenberg model, where 
thcriamous exact solution found by Bethe many years 
agotZI showed that the quantum fluctuations prevent true 
AF LRO, giving instead a slow decay of spin correlations. 
A similar situation is encountered in spin ladders with an 
even number of legs, which , ha\r e a spin gap and purely 
short-range magnetic order .L3Ej This is one of the real- 
izations of a spin-liquid ground state due to purely short- 
range spin correlations. In the limit of a two-dimensional 
(2D) Heisenberg model the spin disorder is replaced by 
a ground state with AF LRO. 

It is well known that frustrated magnetic interactions 
may lead to spin disordered states in two dimensions. 
However, in order to achieve this, i.e., to prevent 2D 
macroscopic spin systems from behaving classically and 
to make quantum mechanics take over instead, the frus- 
tration of the interactions must be sufficiently severe. 
This shows that global SU(2) by itself is not symmet- 
ric enough to defeat classical order in _D > 1 and one has 
to change the magnetic interactions in such a way that 
they lead to sufficiently strong quantum fluctuations. So 



far, this strategy has been shown to lead to spin disorder 
in (quasi) 2D systems in three different situations: (i) 
Frustrating a 2D square lattice by adding longer-range 
AF interactions, as in Ji — J2 and Ji — J2 — J3 mod- 
els, gives a high degeneracy of the classical sector, and 
a disordered state is fewd for particular values of the 
magnetic interactions. CjO This mechanism involves fine- 
tuning of parameters and therefore such systems are hard 
to realize in nature, (ii) In the bilayer Heisenberg model 
two planes are coupled by interlayer AF superexchange 
Jl which generates zero-dimensional fluctuations. This 
leads to a crossover to the disordered ground state of an 
incomptessible spin liquid above a certain critical value 
of Jl .eScj Also this mechanism is hard to realize experi- 
mentally, (iii) In contrast, a spin disordered state can be 
obtained in nature by reducing the number of magnetic 
bonds in a 2D square lattice. The model of CaV409 stud- 
ied by Taniguchi et alrS is a 1/5 depleted square lattice, 
which gives a plaquette resonating valence bond (PRVB) 
ground state for realistic interactions, and ^— ppin gap 
which agrees with experimental observations.Ej A com- 
mon feature of these systems is a crossover between differ- 
ent magnetic ground states, either between two different 
patterns of LRO, as in case (i), or simply between the or- 
dered and disordered states, which results in all three sit- 
uations in a tendency towards the formation of spin sin- 
glets on the bonds with the strongest AF superexchange. 
One may further note that in these spin-only models very 
specific patterns of magnetic interactions are required al- 
ready in two dimensions to prevent the system to order 
classically, while up to now it has proven impossible to 
realize a spin liquid in three dimensions. 

In the present paper we address two fundamental ques- 
tions for the Heisenberg antiferromagnet (HAF) extended 
to include the orbital degrees of freedom in orbitally de- 
generate MHI: (i) Which classical states with magnetic 
LRO do exist in the neighborhood of orbital degener- 
acy? (ii) Are those forms of classical order always stable 
against quantum fiuctuations? We will show that the or- 
bitally degenerate MHI represent a class of systems in 
which spin disorder occurs due to frustration of spin and 
orbital superexchange couplings. This frustration mech- 
anism is different from that operative in pure spin sys- 
tems, and suppresses the magnetic LRO in the ground 
state even in three dimensions. 

As explained above, the low-energy behavior of such 
systems is described by a spin-orbital model. We will 
show that within the framework of such a spin-orbital 
model the occurrence of spin disorder may be regarded 
as resulting from a competition between various classical 
ordered phases, each one with a simultaneous symmetry 
breaking in spin and orbital space. The qualitatively new 
aspect is that the magnetic interactions follow the or- 
bital pattern, and thus these systems tend to " self-tune" 
to (critical) points of high classical degeneneracy. We 
show explicitly that in the vicinity of such a multicritical 
point classical order is highly unstable with respect to 
quantum fluctuations. As a result, a qualitatively new 
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quantum spin liquid with strong orbital correlations is 
expected. We believe that a 3D state of this type is re- 
alized in LiNi02. 

The paper is organized as follows. The spin-orbital 
model for (f transition metal ions, such as Cu^+ in 
KCuFa, is derived in Sec. || using the correct multi- 
plet structure of Cu'^"'" excited configurations. We solve 
this model first in the MP approximation and present the 
resulting classical pha ses and the accompanying orbital 
orderings in Sec. 



The elementary excitations ob- 
tained within an extension of the linear spin- wave theory 
(LSW) are presented in Sec. [V, where we demonstrate 
that two transverse modes are strongly coupled to each 
other. This leads to soft modes next to the classical tran- 
sition lines, and to the collapse of LRO due to diverging 
quantum corrections, as shown in Sec. We summarize 



the results and present our conclusions in Sec. VI 



II. THE SPIN-ORBITAL MODEL 

Our aim is to construct the effective low-energy Hamil- 
tonian for a 3D perovskite-like lattice. The original 
charge-transfer multiband model, as considered for in- 
stance for the cuprates, includes the hybridization ele- 
ments between the 3c? orbitals of |tjp,nsition metal ions 
and the 2p orbitals of oxygen ionsH3 If the Coulomb el- 
ements at the 3(i orbitals and the charge-transfer energy 
between the 3c? and 2p orbitals are large, this model can 
be transformed into an effective spin-fermion model. For 
example, this transformation performed for the three- 
band model gives an effective Hamiltonian with localized 
spins at the Cu sites which interact by superexchange 
interactions, while the doped carriers interact with them 
by a Kondo-like coupling.c3 In the limit of undoped com- 
pounds, one is thus left with a model which describes 
interacting transition metal ions. 

The simplest form of (superexchange) interaction, 
namely a spin model, is obtained for the case of non- 
degenerate c? orbitals, whereas orbital degeneracy gives 
a spin-orbital model acting in a larger Hilbert space de- 
fined by both spin and orbital degrees of freedom at each 
transition metal site. Having in mind the strongly corre- 
lated late transition metal oxides, we consider specifically 
the case of one hole per unit cell in the 3d^ configuration, 
characterized in the absence of JT-distortion by two de- 



generate Cg orbitals: cc — y ~ \x) and (3z 



\z). The derivation is, however, more general and ap- 
plies as well to the low-spin c?^ configuration; in the case 
of the early transition metal oxides the case would 
involve the t2g orbitals instead. 

The holes in the undoped compound which corre- 
sponds to the d^ configuration of transition metal ions, as 
in La2Cu04 or KCuFa, are fairly localizedo Hence, we 
take as a starting point the following Hamiltonian which 
describes c?-holes on transition metal ions. 



and consider the kinetic energy, H^in, and the electron- 
electron interactions. Hint: within the subspace of the Cg 
orbitals (the t2g orbitals are filled by electrons, do not 
couple to Cg orbitals due to the hoppings via oxygens, 
and hence can be neglected) . The last term describes 
the crystal-field splitting of the eg orbitals. 

Due to the shape of the two eg orbitals \x) and \z), 
their d — p hybridization in the three cubic directions is 
unequal, and is different between them, so that the effec- 
tive hopping elements are direction dependent and differ- 
ent for \x) and \z). The only nonvanishing hopping in the 
c-direction connects two \z) orbitals, while the eleraepts 
in the (a, b) planes fulfill the Slater-Koster relations ,c3 as 
presented before by two of us.E£l Taking the hopping t 
along the c-axis as a unit, the kinetic energy is given by. 



Hhin. 



+ H.c.) + dlj^ +t^Y. 4zA..^ (2-2) 

where and (ij)-L stand for the bonds between near- 
est neighbors within the (a, 6)-planes, and along the c- 
axis, respectively, and /3 = 1 in a cubic system. The 
X — z hopping in the (a, b) planes depends on the phases 
of the x^ — orbitals along a- and 6-axis, respectively. 
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H, 



(2.1) 



included in the factors (-l)'''^ in Eq. (|^. 

The electron-electron interactions are described by the 
on-site terms, 

Hint= (U + \ Jh) ^ nia^^Tlial + [U - Jr) ^ n.^^Uiza 
ia icr 

+ (U ~ \Jh) n,^„n,zs - \Jh 4xA^a4zAza 
ia ia 

+ l^H^idl^^dl^AzAz] +4z^4zidixidtx]), (2-3) 



with U and Jh standing jfor the Coulomb and Hund's 
rule exchange interaction,EJ respectively, and a = x, z. 
For convenience, we used the simplified notation a — —a. 
This Hamiltonian describes correctly the multiplet struc- 
ture of d^ (and ci^)rians,E3 and is rotationally invariant 
in the orbital space.Eil The wave functions have been as- 
sumed to be real which gives the same element Jh/2 for 
the exchange interaction and for the pair hopping term 
between the eg orbitals, and \z). 

In fact, we adopted here the most natural units for the 
elements of the Coulomb interaction, with the energy of 
the central \^E) doublet being equal to U. By definition 
this energy does not depend on the Hund's exchange el- 
ement Jh, as we show below, and is thus the measure 
of the average excitation energy in the c?,^d^ dj^d^ 
transition. The interaction element Jh stands for the 
singlet-triplet splitting in the d^ spectrum (Fig. |^) and 
is just twice as big as the exjckange element K^z used 
usually in quantum chemistryBS The typical energies for 
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the Coulomb and exchange elements can be found using 
constrained-occupation local-density functional theoryS 
Unfortunately, such calculations have been performed 
only for a few compounds so far. For La2Cu04, a par- 
ent compound of superconducting cuprates, one finds 
U = 7.77 eV and Jh = 2.38 eV;El other estimations of U 
based on the experimental data-report values 6 < C/ < 8 
eV for cuprates and nickelatesO This results in the ratio 
Jh/U ~ 0.3 which we take as a representative value for 
the strongly correlated late transition metal oxides. The 
values of intersite hopping t, being an effective parameter, 
are more difficult to estimate. As a representative value 
for La2Cu04 one might take t « 0.65 eV, which results 
in the superexchange interaction between the jx^ orbitals 
in (a, 6) planes, J(a,b) = {^l'^)t^lU ~ 0.13 eVj^j in good 
agreement with the experimental value. c3 Similar values 
of the effective t are expected also in the other transi- 
tion metal oxides, and thus we can safely assume that 
at th e filling of one hole per ion the ionic Hamiltonian 
( ^.l[ ) describes an insulating state, and that the effec- 
tive magnetic interactions can be derived in the strongly 
correlated regime of t <C J7. 



The last term in Eq. (2.1) stands for the crystal field 
which lifts the degeneracy of the two Cg orbitals and 
breaks the symmetry in the orbital space. 



(2.4) 



if £x 7^ It acts as a magnetic field in the or bita l space, 
and together with the parameter /3 in Hkin (2^) quan- 
tifies the deviation in the electronic structure from the 
ideal cubic local point group. 

In the atomic limit, i.e., at t = and Ez = 0, one has 
orbital degeneracy next to spin degeneracy. This gives 
four basis states per site, as each hole may occupy either 
orbital, \x) or \z), and either spin state, a =\ or a =1. 
The system of N (P ions thus has a large degeneracy 
4^, which is, however, removed by the effective interac- 
tions between each pair of nearest neighbor ions 
which originate from virtual transitions to the excited 
states, d^d^ ^ dj^dj, due to hole hopping. Hence, we de- 
rive the effective spin-orbital model following Kugel and 
Khomskiijj starting from the Hamiltonian in the atomic 
limit. Hat = Hint + H^, and treating Hkin as a pertur- 
bation. However, in the present study we include the 
full multiplet structure of the excited states within the 
configuration which gives corrections of the order of 
Jh compared with the earlier results of Refs. |^,^. 

Knowing the multiplet structure of the (P intermedi- 
ate states, the derivation of the effective Hamiltonian can 
be done in various ways. The most straightforward but 
lengthy procedure is a generalization of the canonical 
transformation methad used before for the HubbardO 
and the three-bandcj model. A significantly shorter 
derivation is possible, however, using the cubic symme- 
try and starting with the interactions along the c-axis. 
Here the derivation simplifies tremendously as one finds 



only effective interactions which result from the hopping 
of holes between the directional \z) orbitals, as shown 
in Fig. |l| Next the interactions in the remaining direc- 
tions can be generated by the appropriate rotations to 
the other cubic axes a and 6, and applying the symmetoi 
rules for the hopping elements between the Cg orbitals.cEl 
The derivation of the spin-orbital model is given in more 
detail in Appendix 

Depending on whether the initial state is |z)i|a;)j or 
\z)i\z)j, the intermediate d}'^d^ configuration resulting 
from the hole- hop \z)i ^ involves on the site ei- 
ther the interorbital states, the triplet and the singlet 
^Eg, or the two singlets built from the states with doubly 
occupied orbitals, ^E^ and ^Ai. Of course, the spins have 
to be opposite in the latter case, while in the former case 
also parallel spin configurations contribute in the triplet 
channel. Apart from a constant term, this atomic prob- 
lem is equivalent to that of the configuration, and 
thus one might consider instead the spectrum of d^ ions. 
The eigenstates within the eg subspace are: (i) triplet 
[^^2), (ii) interorbital singlet \^E^), and (iii) bonding and 
antibonding singlets, \^Eg) and with double oc- 

cupancies of both orbitals, where bonding/antibonding 
refers to pair hopping term oc Jh between \x) and \z) 
orbital. The energies of the states ^^2) and l^E^) are 
straighforwardly obtained using Six • Siz = +1/4 and 
Six • Siz = —3/4, for S* = 1 and 5 = states, respec- 
tively. The remaining two singlet energies are found by 
diagonalizing a 2 x 2 problem in the subspace of doubly 
occupied states. Hence, the resulting spectrum is,EZl 



^Jh — ■^Jh — (Ez/Jh)^] ^ , 

yH + ^,JH[l-iEz/JHff\ (2.5) 

At Ez ~ it consists of equidis- 
stance of Jh between the triplet 
PA2) and the degenerate singlets \^Eg) and \^E^) (which 
form of course an orbital doublet), as well as between the 
above singlets and the top singlet We emphasize 

that the simp lifie d Hubbard-like form of electron-electron 
interactions ( ^.3| ) which uses two parameters, U and Jh, 
in this case is an exact representation of the Coulomb 
interaction in the ifg^g configuration as obtained in the 
theory of multiplet spectra, and one finds a one-to-one 
correspondence between the energies calculated above, 
and-those found with the Racah parameters A, B, and 

c,eI 

E{^A2)=A-8B, 



ECA2) = 


U-Jh, 


E^E,) = 




ECEe) = 


U+\Jh 


ECA,) = 


U+^Jh 


where Ez — 


£x - £z- 


tant states. 


with a c 



eCe) =A + 2C, 
ECAi) = A + 8B + 4C. 



(2.6) 



Thus, the parameters used by us are U = A + 2C and 
Jh — S>B + 2CJ13 We normalize the energies by the 
Coulomb interaction U, and introduce 



5 



V = Jh/u 



(2.7) 



as an energy unit for the Hund's rule exchange interac- 
tion. This gives the excitation energies which correspond 
to the local excitations dfd3 dj^d^ on a given bond 



V, 



eCEg) = 1 + iry - i,7 [1 - (i^JJ^)'] 



e('Ai) = l 



1/2 



1/2 



(2.8) 



shown in Fig. g. We note that the deviation from the 
equidistant spectrum at i?^ = becomes significant only 
for \Ez\/Jif > 1. Taking the realistic parameters of the 
cupratesj^ one finds for La2Cu04 with = 0.64 eV 
that Ez/Jh — 0.54, a value representative for systems 
that are already far from orbital degeneracy. Since we 
are interested here in what happens close to orbital de- 
generacy, this allows us to neglect the E^ dependence of 
the energie s of the excited states, and use the atomic 
spectrum (2.6) in the derivation presented in Appendix 
@. 

Following the above procedure, we have derived the 
effective Hamiltonian Ti. in spin-orbital space, 



7i — "Hj + Ti-T, 



(2.9) 



where the superexchange part Ti.j can be most gener- 
ally written as follows (a simplified form was discussed 
recently in Ref. ^), 



E 



Si • Sj -\ - 



ei'E,) 
t^ 



Si • S' 



ei^Eg) e(iAi) 



Si • Sj 



■pa 

' {ij) 



-pa 



(2.10) 



Here Si refers to a spin 5 = 1/2 at site z, and V^fj^ are 
projection operators on the orbital states for each bond. 



V 



a 



2(|-rf)(i-rn. 



They are either parallel [Pi^ — ^ — rf ) to the direction of 
the bond (ij) on site i, and perpendicular {Pj^ ~ 5 + '^j ) 
on the other site j, or parallel on both sites, respectively, 
and are constructed with the following orbital operators 
associated with the three cubic axes (a, 6, c). 



The (t's are Pauli matrices acting on the orbital pseudo- 
spins |x) = ^ J ^ , \z) = (^1^. Hence, we find a Heisen- 

berg Hamiltonian for the spins, coupled into an orbital 
problem. While the spin problem is described by the con- 
tinuous symmetry group SU{2), the orbital problem is 
clock-model-like, i.e., there are three directional orbitals: 



' , but they are not indepen- 



dent. The orbital basis consists of one directional orbital 
and its orthogonal counterpart, and we have chosen here 
\z) = 3z^ — and \x) = — orbitals. 

In general, the energies of these two orbital states, \x) 
and jz), are different, and thus the complete effective 
Hamiltonian of th e model (2.9) includes as well the 
crystal-field term (^.4[) which we write as 



Hr = -E, 



(2.13) 



Here E^ is a crystal field which acts as a "magnetic field" 
for the orbital pseudospins, and is loosely associated with 
an uniaxial pressure along the c-axis. The d^ spin-orbital 
model ( 2.9| ) depends thus on two parameters: (i) the crys- 
tal field splitting E^, and (ii) the Hund's rule exchange 
Jh- 

While the first two terms in ( 2.10| ) cancel for the mag- 
netic interactions in the limit of 77 — )■ 0, the last term 
favors AF spin orientation. Although the form ( 2.10| ) 
might in principle be used for further analysis, we pre- 
fer to make an expansion of the excitation energies e„ 
in the denominators of Eq. ( 2.10| ) in terms of Jh, and 
use rj ~ Jh/U (2.7) as a parameter which quantifies the 
Hund's rule exchange. This results in the following form 
oft he, effective exchange Hamiltonian in the d^ model 

(HlM 



2{S,- S. - 



7CC 



Ipa 



4' (u) 



(2.14) 



The first term in Eq. (2.14) describes the AF superex- 
change cx J — t^ jU (where t is the hopping between 
\z) orbitals along the c-axis), and is obtained when the 

(2.11) splittings between different excited states ^ Jh (Fig. 
|) are neglected. As we show below, in spite of the AF 
superexchange oc J, no LRO can stab ilize in a system de- 
scribed by the spin-orbital model ^■!\ ) in the limit 77 — > 
at orbital degeneracy {E^ — 0) because of the presence of 
the frustrating orbital interactions which gives a highly 
degenerate classical ground state. We emphasize that 
even in the limit of Jh the present Kugel-Khomskii 
model does not obey SU(4) symmetry, essentially because 
of the directionality of the eg orbitals. Therefore, such an 
idealized SU(4)-symmetric modelEj does not correspond 

(2.12) to the realistic situation of degenerate eg orbitals and is 
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expected to give different answers concerning the inter- 
play of spin and orbital ordering in cubic crystals. 

Taking into ac coun t the multiplet splittings, we obtain 
[second line of ( 2.14 )] again a Heisenberg-like Hamilto- 
nian for the spins coupled into an orbital problem, with 
a reduced interaction oc Jij. It is evident that the new 
terms support FM rather than AF spin interactions for 
particular orbital orderings. This net FM superexchange 
originates from the virtual transitions which involve the 
triplet state ^^2), which has the lowest energy and thus 
gives the strongest effective coupling. We remark in pass- 
ing that the FM channel is additionally enhanced for d* 
ions when the virtual excitations to double occupancies 
in eg orbitals happen in the presence oLpartly filled t2g 



orbitals, as realized in the manganites.ESi 

The important feature of the spin-orbital model (2.£) 
is that the actual magnetic interactions depend on the or- 
bital pattern. This follow s es sentially from the hopping 
matrix elements in (2.2) being different between a 

pair of \x) orbitals, between a pair of different orbitals 
(one \x) and one \z) orbital), and between a pair of \z) 
orbitals, respectively, and depending on the bond [direc- 
tion either in the (a, b) planes, or along the c-axis.t2l We 
show in Sec. Ill that this leads to a particular competi- 



tion between magnetic and orbital interactions, and the 
resulting phase diagram contains a rather large number 
of classical phases, stabilized for different values of 
and Jh- 



III. MEAN-FIELD PHASE DIAGRAM 

A. Anisotropy of antiferromagnetic interactions 

We start the analysis of the spin-orbital (or Kugel- 
Khomskii) model (2.£)-(2.14) by analyzing the MF so- 



lution obtained by replacing the scalar products Si ■ Sj 
by the Ising term, SfS^. The MF Hamiltonian may be 
written for the more general situation where the interac- 
tion has uniaxial anisotropy along the c-direction in the 
3D lattice as follows. 



7« 



-T] 



cc 



19« 



7« 



4 ' (y) 



(ij) 



(3.1) 



where — Jb = J, and Jc = J/3. For (3 > 1 the nearest- 
neighbor bonds (ij) \\ c are shorter, while for (3 < 1 these 
bonds are longer than the bonds within the (a, 6) planes. 
In the limit of /3 ^ the bonds along the c-axis may be 
neglected and the model reduces to a 2D model, repre- 
sentative for the magnetic interactions between Cu ions 
within the CUO2 planes of the high-temperature super- 
conductors. 



The presence of AF spin interactions cx J suggests 
magnetic superstructures with staggered magnetization, 
and we considered several possibilities, with two- and 
four-sublattice 3D structures, giving rise to G-AF and 
A-AF phases, AF ID chains coupled ferromagnetically, 
and others. The MF Hamiltonian contains as well an AF 
interaction between orbital variables, ^ jT°^Tj', which 
suggests that it might be energetically more favorable to 
alternate the orbitals in a certain regime of parameters, 
and pay thereby part of the magnetic energy. This il- 
lustrates the essence of the frustration of the magn etic 
interactions present in the spin-orbital model (2.9), as 
discussed in Sec. |. Therefore, for any classical state the 
orbitals occupied by the holes have to be optimized, and 
we allowed mixed orbitals (MO), 



\i^l(7) 



(3.2) 



with the values of the mixing angles {9i\ being varia- 
tional parameters to be found from the minimization of 
the classical energy. 



The superexchange in (3T) depends strongly on the or- 
bital state. At large positive E^, where the crystal field 
strongly favors |x)-occupancy over |z)-occupancy, one ex- 
pects that 9i — 7r/2 in Eq. (3.2), and the holes occupy 
\x) orbitals on every site. In this case the spins do not 
interact in the c-direction (see Fig. |l|), and there is also 
no orbital energy contribution. Hence, the (a, b) planes 
will decouple magnetically, while within each plane the 
superexchange is AF and equal to 9 J/4 along a and b. 
These interactions stabilize a 2D antiferromagnet, called 
further AFxx. The resulting 2D Neel state with decou- 
pled (a, b) planes along the c-direction is the well-known 
classical ground state of the high-Tc superconductors, 
La2Cu04 and YBa2Cu3 06.E3 In contrast, if i?^ < and 
\E^\ is large, \E^\/ J > 1, then 9, = in Eq. (|^), 
and the holes occupy \z) orbitals. The spin system has 
then strongly anisotropic AF superexchange, being 4J 
between two \z) orbitals along the c-axis, and J/4 be- 
tween two \z) orbitals in the (a, 6) planes, respectively. 
The corresponding 3D Neel state with holes occupying 
\z) orbitals is called AFzz. The spin and orbital order in 
both AF phases is shown schematically within the (a, b) 
planes in Fig. 0. 



B. Antiferromagnetic states in the 3D model 

Assuming an AF classical order in all three directions, 
the so-called G-AF state, it is thus obvious that for large 
\Ez\ one finds either the AFxx or the AFzz phase, de- 
pending on whether E^ > 01 E^ < 0, with the following 
energies normalized per one site. 



£^AFxx — 
EaFzz = 



-J 1 



(3.3) 
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The AFxx and AFzz phases are degenerate in a 3D sys- 
tem (/3 = 1) along the Hne Ez = 0, while decreasing /3 
moves the degeneracy to negative values of E^, namely 
to£;, = -2J(l- /?)(!- 2). 

However, for intermediate values of \Ez \ one should al- 
low for mixed orbitals. Following the argument above 
about the AF nature of the orbital interaction, we as- 
sume alternating orbitals at two sublattices, A and B. 
The alternation should allow the orbitals to compromise 
between being identical (optimizing the magnetic energy) 
and being orthogonal (optimizing the orbital energy). 
This is realized by choosing in Eq. (3.2) the angles alter- 
nating between the sublattices: 9i — +6 for i G A, and 
6j = —9 for j £ B, respectively, 



ji/ifj) = cos 9\iza) + sin 9\ixa) ^ 
Ij/ifj) = cos 9\jza) — siTi9\jxa). 



(3.4) 



The calculation of the energy can be performed either by 
evaluating the average values of the operator variables 
{r"}, or by taking the average values of the orbital pro- 
jection operators {Pia} as given in Eq. (A3). Using the 
two-sublattice orbital ordering (3.4), one finds for the 
bonds (ij) \\ (a, 5), 



{P^^PjC + P^iPji) = 1(7 - 4cos2 29), 
{2P,cPjc) - i(l-2cos20)2, 



and for the bonds (ij) 



"f" PizPjx) 

m.Pj.) 



i(l-cos^ 29), 
i(H-cos26i)2. 



(3.5) 



(3.6) 



The classical energy per site as a function of 9 is then 
given by 

£;(0) = -^(l + |)(7-4cos2 2^?) 



J 



^(l-^)(l-2cos2^^)2 



E, cos 29. 



cos^ 29) 
cos 20)2 



This has a minimum at 

(l-i)(l-/3) 



cos 26* 



(2 + /3)77 



(3.7) 



(3.8) 



where = E'^/ J, if r/ 7^ 0, and provided that | cos 29\<1 
(a similar condition applies to all the other states with 
MO considered below). So, as long as 2 J(/3 — 1) — 3 J(/3-|- 
1)?? <Ez< 2J{j3 -l) + J(5 + f3)r], there is genuine MO 
order, while upon reaching the smaller (larger) bound- 
ary value for Ez, the orbitals go over smoothly into \z) 



{\x)), i.e. one retrieves the AFzz (AFxx) phase. Taking 
the magnetic ordering in the three cubic directions \ab c] 
as a label to classify the classical phases with MO ([3.4|), 
we call the phase obtained in the regime of genuine MO 
order MOaaa, with classical energy given by 



E, 



MOAAA 



= - 2 



J 



[{2-rj){l-l3)+ezY 
4(2 + /3)77 



(3.9) 



Upon increasing Jh, the FM interactions occur which in- 
crease the energy of the AF phases in three dimensions 
by the term ^rj per site in Eqs. ( |3.3| ) (a similar increase 
of energy occurs also in the MOaaa phase in the region 
of its existence). This indicates frustration of magnetic 
interactions and opens a potential possibility that other 
classical phases with FM order along particular direc- 
tions might be more stable. We have found a few classi- 
cal phases when the spins order ferromagnetically either 
in particular planes, or along one spatial direction, and 
this magnetic order coexists with M O oc cupied by holes. 

For example, the angles in Eq. (3^) can be chosen 
in such a way that at least one of the orbitals on two 
neighboring sites is perpendicular to the bond direction, 
e.g. is like — type for a bond along the a-axis. 
In such a case, the AF superexchange vanishes, and one 
finds instead a weaker FM interaction, in agreement with 
the Goodenough-Kanamori rules E3 By this mechanism 
Kugel and Khomskii3 proposed an alternating orbital or- 
der to explain the FM planes observed in KCuFs. Follow- 
ing this argument, let us assume FM order within (a, b) 
planes, and the same form (3.4) as above for the alter- 
nating orbitals at the two sublattices, A and B. As alter- 
nating orbitals can only be arranged to be perpendicular 
to the bonds in at most two spatial directions, such an 
arrangement for the (a, b) planes forces the orbitals to 
have nonzero lobes along c. This results in sizable AF 
superexchange for the bonds (ij) parallel to c, which will 
order the spins antiferromagnetically in the c direction. 
The orbitals may either repeat or stagger along the c- 
axis, and both states give the same mean-field energy. 
Taking the magnetic ordering in the three cubic direc- 
tions lobe] as a label to classify the classical phases with 
MO ( |3.4[ ), we call this ground stat e th e MOffa phase. 
With the help of Eqs. (3.5) and (3.6) one obtains the 
following classical energy as a function of 9, 



E{9) = --^(l + 77)(7-4cos2 261) 
-^/3(l + f)(l-cos2 2^?) 
-^/3(1-|)(1 + cos 20)2 
-I- -Ez cos29. 



(3.10) 



with a minimum at 
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COS 20 



(2 + /3)77 



(3.11) 



where agai n the MO exist as long as | cos20| < 1. Using 
Eqs. ( M0|) and §1^) one finds that the classical energy 
of the MOffa phase is given by 



-Bmoffa 



J 



(11-777) 



2 2 



1 



if? 1 = 



(2 + /3)r7 



(3.12) 



As a special case, let us consider first dege nera te or- 
bitals {E^ = 0) in a 3D system {(3=1). Eq. ( ^.ll[ ) sim- 
plifies in this case to cos26' = (1— ^)/(2 + 37;). A partic- 
ularly simple result is found at 77 = where cos 20 = 1/2, 



? 2 

y -z 



7r/6, and the orbitals stagger like x 



and 



, as shown in Fig. ^ This staggering was proposed 
by Kugel and Khomskii as a ground state of KCuFsp of 
course, this state is not realized for the realistic parame- 
ters with 77 ~ 0.3, but the optimized orbitals with 9 given 
by (3.11) are not so far from this idealized picture. 

The energy of the MOffa phase is degenerate with 
that of the AF phases at the classical degeneracy point, 
M = {Ez/J,ri) = (0,0), and this phase becomes more 
stable at ?7 > 0, and Ez/J — 0. The magnetic en- 
ergy is gained due to relatively strong AF interactions 
on the bonds (ij) \\ c, and weak FM interactions in the 
planes (a, b) , pe rpe ndicular to the preferred directional- 
ity of the MO (3.2) along the c-direction, while the or- 
bital energy is gained due to orbital alternation within 
the (a, 5) planes. Such orbital ordering remains stable 
with decreasing Ez < 0, while two similar states with the 
staggering either within the (6, c) or the (a, c) planes, are 
more stable for Ez > 0. Following our convention, these 
two degenerate MO states stable at > are called 
MOaff and MOfaf (see Fig. respectively. However, 
the MO involve in this case the directional orbital |C) 
along the AF bonds (i.e., \(a) 3x^ — for MOaff or 
ICft) ~ 3?/^ — for MOfaf, respectively), and the cor- 
responding orthogonal orbital, |^). Therefore, since the 
symmetry-breaking field acts on \z) orbitals, the angles 
in the two sublattices cannot be exactly equivalent in 
this case, unlike in the MOffa phase, and we adopted an 
ansatz, 



\ia) = cos 6*+ 1 z^cr) + sin 6 +\i(a), 
\ja) = cos 9^\i£^a) — sm9-\i(iT), 



(3.13) 



where i € A, j e B, and 9± > for the two sub- 
lattices. Introducing for convenience the new angles, 
(t> = ^ {S+ +9-), and 6 = 9+ — 6-, one finds the follow- 
ing conditions for the energy minimum of the classical 
MOaff phase, 

cos 2(/> = {[(1 -I- /3)(2 - 77) -f cos 5 

+ V3ezSmS^[l + (3+{l + 2P)ijr\ (3.14) 
tan 2(5 = +^V3 [(1 + /3)(2 - 77) + Bz] Sz 

X {4 [1 + /3 + (1 + 2/3)77] + [(1 + m - V) + ez? 

-lel]-\ (3.15) 



and the energy is given by 
J 



E 



MOAFF 



[7(1 + 77) + 2/3(1 + COS ,5)] 



J [[(l + /3)(2-77)+e^]cos(5 + VSe^sinJ]' 
32 



1 + /3 + (1 + 2/3)77 



(3.16) 



Finally, one may consider states in which magnetic 
energy is gained in the c-direction due to MO with a 
small admixture of \z) into orbitals of p redominantly 
character, i.e., sin^i = 1 — e in Eq. (3^). As such a state 
is a modification of the AFxx phase, the two sublattices 
in the (a, b) planes are again physically equivalent, and it 
suffices to introduce a single angle 9 to characterize this 
state. Apart from (large) energy contributions due to AF 
order on the bonds in the (a, b) planes, the expansion of 
the ground state energy contains also (small) terms de- 
pending on the spin order in the c-direction, (S'fS'|)||c, 

E={1 + 00826*) (1 + cos 20 - 7/) (S'f + const, 

(3.17) 

which prefers FM order as long as (1 + 008 20) < rj. The 
reason is that the AF superexchange is a fourth order 
effect ^ e^, while the FM interactions oc 77 are second 
order, ~ e^, and give a lower energy E as long as the \z) 
occupancy is small enough. Following our convention, 
we call the resulting state the MOaaf phase, with the 
mixing angle given by 



cos 20 : 



1 - 2 
^ 2 



/3(l + 77) + 277 ' 



(3.18) 



and the classical energy by 



£^MOAAF = - ( 2 + -7/ ) J 



(3.19) 



Therefore, only when the average population of the \z) 
orbitals, ~ cos^ 0, increases sufficiently, one can find a 
transition to the AF phase with mixed orbitals, MOaaa, 
discussed above. 

By making several other choices of orbital mixing and 
classical magnetic order, we have verified that no other 
commensurate ordering with up to four sublattices can 
be stable in the present situation. Although some other 
phases could be found, they were degenerate with the 
above phases only at the M point, and otherwise had 
higher energies. Thus, we obtain the classical phase di- 
agram of the 3D spin-orbital model (2.9) by comparing 
the energies of the six above phases for various values 
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of two parameters, {E^/ J, Jh/U}: two AF phases with 
two sublattices and pure orbital character (AFxx and 
AFzz), three A-AF phases with four sublattices (MOffa 
and two degenerate phases: MOaff and MOaff), one 
C-AF phase (MOaaf), and one G-AF phase with MO's 
(MOaaa). While the orbital mixing is unstable at 77 = 0, 
the generic sequence of classical phases at finite rj and 
decreasing E^/J is: AFxx, MOaaf, MOaaa, MOaff, 
MOffa, and AFzz, and the magnetic order is tuned to- 
gether with the gradually increasing \z) character of the 
occupied orbitals. 

The result for cubic symmetry {P — 1) is presented in 
Fig. 1^, where one finds all six phases, but the MOaaa 
phase does stabilize only in a very restricted regime of 
parameters with Jh/U < 0.1, before MOaff takes over. 
Only the first of the above transitions is a continuous 
one, and the \z) amplitude ~ cos^ 9 increases smoothly 
from zero and removes the built-in degeneracy of the 2D 
AFxx phase with respect to the magnetic order along 
the c-direction. All the other transition lines in Fig. H 
are associated with jumps in the magnetic and in orbital 
patterns. We emphasize that all the considered phases 
with magnetic LRO are degenerate at the point M, with 
classical energy of — 3J. In fact M is an infinite-order 
quantum critical point, since not only may the spins be 
chosen to be FM in certain planes, whence the orbitals 
have to be tuned to compensate the loss of the magnetic 
energy by the orbital energy contributions, as realized 
in all MO phases, but also may the orbitals be rotated 
freely when the spins are AF in all three directions. We 
note, however, that the magnetic terms are essential, and 



in a purely disordered spin system, with {SfS^ 



0, 



higher energy of — 2U/8 is found even with the optimal 
choice of orbitals with cos 20 = 0. 

We also investigated the phase diagrams for the case 
of modified hopping along the c-direction (/3 7^ 1). One 
finds that increased hopping {(3 = 1.414) in the c- 
direction stabilizes the MO phases, and in particular the 
MOaff (MOfaf) phase [Fig. ||(a)]. By contrast, the 
MO phases are stable in a narrower range of E^ for a 
fixed value of Jh /U , if the hopping along the c-direction 
is decreased below /? = 1 [an example of /3 = 0.707 
is shown in Fig. ||(b)]. The decreased stability of the 
MOaff phase promotes in this case the AF order with 
MO in the MOaaa phase. The latter phase is stable 
only in a relatively narrow range of Ez^ and only for 
small enough Jh/U; an increase of Jh/U favors instead 
FM order along the c-direction. We also n ote that the 
orbital mixing sets in for the MOaaa phase (3^) o nly a t 



a smaller value of E^ than in the MOaaf phase ( p.l8[ ). 
Interestingly, the point of high degeneracy of the classical 
states exists independently of the value of (3, and moves 
for /3 7^ 1 to E,_ = -2J(1 - /3). This demonstrates the 
generic nature of the internal frustration of spin and or- 
bital interactions in the model, and the crystal field term 
just plays here a compensating role for the missing (or 
enhanced) magnetic interactions within the (a, 6) planes. 
Independently of the value of /?, the spin-orbital model 



^ has a universal feature: different classical spin struc- 
tures become degenerate at the critical lines in Figs. ^j^. 
This is also encountered in frustrated 2D magnetic lat- 
tices described by simple Heisenberg Hamiltoniansjlj and 
may thus be regarded as a signature of frustration. How- 
ever, unlike in the purely spin models, in the present case 
( ^.9[ ), the sign of the interactions changes because of the 
coupling to the orbital sector, and this reduces the effec- 
tive dimensionality for the AF interactions ~ J, with the 
3D system behaving like a quasi-lD antiferromagnet. 



C. Phase diagram of a 2D model 



As a special case, we considered the limit of /3 — *■ 
which gives a 2D spin-orbital model. The two AF phases 
with either \x) or \z) orbitals occupied, AFxx and AFzz, 
are degenerate at E^ = —2J. This asymmetry reflects 
the large difference between the superexchange interac- 
tions for \x) and \z) orbitals within the (a, b) planes of a 
2D syst em w hich has to be compensated by the orbital 
energy ( 2.13 ). 

As the presence of FM planes || c-axis is crucial for the 
ordering in the MOaff phase (see Fig. H), this phase dis- 
appears, while the remaining two phases with AF order 
within (a, b) planes, MOaaa and MOaaf, collapse into a 
single MOaa phase. Hence, one finds in two dimensions a 
classical phase diagram with only four phases, which are 
stable with decreasing E^ and at finite rj in the follow- 
ing order: AFxx, MOaa, MOff, and AFzz (Fig. |). The 
2D phase diagram shows in particular that strong AF su- 
perexchange in the c-direction is not the stabilizing factor 
of the MOffa phase in the 3D model, but instead these 
phases are stable due to the orbital interactions which 
enforce the orbital alternation shown in Fig. ^ 

For the realistic parameters of La2Cu04 the Cu|-6L2_y2 
and d^z^-r^ orbitals are split, and E^ ~ 0.64 eVJB This 
material belongs together with Nd2Cu04 to the class of 
cuprates with weakly coupled Cu02 planes, and one finds 
in the present treatmeat a 2D AFxx state, as observed in 
neutron experiments.Eil If however the orbital splitting 
is small in a 2D situation, the orbital ordering couples 
strongly to the lattice, as the hybrids with alternating 
phasing on two sublattices are formed according to Eqs. 
( p^ The net result is a quadrupolar distortion as indi- 
cated in Fig. 0. In fapt, using these arguments Kugel and 
Khomskii predictecH the existence of such a structural 
distortion in the MOff phase of a quasi-2D compound 
K2CUF4. This prediction was confirmed experimentally 
a few years later .e2I 

The MOff phase of K2CUF4 is magnetically polarized, 
has no quantum fluctuations, and is thus well described 
in a classical theory. In the next sections we concentrate 
ourselves on the 3D case, where the quantum fluctuations 
are strong and destabilize the classical magnetic ordering 
in a particular regime of parameters. 
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IV. ELEMENTARY EXCITATIONS 
A. General formalism 

The presen ce o f the orbital degrees of freedom in the 
Hamiltonian (2.9) results in excitation spectra that are 
qualitatively different from those of the HAF with a sin- 
gle spin-wave mode. As we have discussed in the limit 
of Jh = 0, the transverse excitations are twofold: spin- 
waves and spin- and- orbital waves In addition to these 
two modes there are also longitudinal (purely orbital) ex- 
citations, and thus one finds three e]£meata«2-|excitations 
for the present spin-orbital model ( 2^ ) .OS^a This gives 
therefore the same number of modes as found in a ID 
SU(4) syaiHietric spin-orbital model in the Bethe ansatz 
method.L^E^ We emphasize that this feature is a conse- 
quence of the dimension (equal to 15) of the so(4) Lie 
algebra of the local operators, as explained below, and 
is not related to the global symmetry of the Hamilto- 
nian. Here we present the analysis of the realistic d^ spin- 
orbital model for the 3D simple cubic (i.ep-|perovskite- 
like) lattice, using linear spin-wave theory£Ji£II general- 
ized such as to make it applicable to the present situation. 

Before we introduce the excitation oper ato rs, it is con- 
venient to rewrite the spin-orbital model (2^) in a differ- 
ent representation which uses a four-dimensional space, 
{|a;t), l^^i), 1-2 T)i 1-2 i)}, instead of a direct product of 
the spin and orbital spaces. Hence, we introduce oper- 
ators which define purely spin excitations in individual 
orbitals, 



SL^dl^d,,^, (4.1) 



and operators for simultaneous spin-and-orbital excita- 
tions, 

KL = dlx^d.zi^ K+x = 4T^.xi- (4-2) 

The corresponding Sf^^ and K^^^ operators are defined 
as follows. 



qz _ 

ixx 


qij^ix^ '^ixX)j 




izz 


^("-izT - 


(4.3) 


= 

IXZ 






= 

IZX 


hi^izTdix^ — dl^idixi)- 


(4.4) 



The Hamiltonian (2.£) contains also purely orbital in- 
teractions which can be expressed using the following 
orbital-fiip {Tiap) and orbital-polarization (ni_) opera- 



W\x^d^^^+4xid^zl), 



tors. 




Tixz 


= Wl 


T 

J- tzx 


-Wlz 




-Wlx 



+ 4zldtxl), 



In order to simplify the notation, we also introduce sum 
operators for the spin-and-orbital and purely orbital op- 
erators. 



I ixz ' %zx^ 



T — T- 



(4.6) 



- \{d\x^d^x^+d\^^d,xl-~d\^^d,^^-d\^^di^l). (4.5) 



'iz] 



izl 



The full set of local operators at a site i co nstit ute an 
so(4) Lie algebra. While the spin operators ([4.l| ) fulfill 
of course for x and z separately the usual su{2) commu- 
tation relations, they also form collectively a subalgebra 
of so(4), and the same holds for the spin-and-orbital op- 
erators ( |4.2| ). However, as we will see below, for the cal- 
culation of the excitations one also needs commutators 
between spin and spin-and-orbital operators, so that one 
cannot avoid considering the full Lie-algebra structure of 
so(4) , discussed in Appendix 

The number of collective modes in a particular phase 
may be determined as follows. The so(4) Lie algebra con- 
sists of three Cartan operators, i.e., operators diagonal on 
the local eigenstates of the symmetry-broken phase un- 
der consideration (e.g. S^^^, ^izz^ ^'^d rit- in the AFxx 
phase), plus 12 non-diagonal operators turning the eigen- 
states into one another (like Sf^^ and Sf^^ in AFxx). 
Out of those twelve operators, six connect two excited 
states (like S^^^ in AFxx), and are physically irrelevant 
(at the RPA level), because they give only rise to 'ghost' 
modes, modes for which the spectral function vanishes 
identically. The remaining six operators connect the lo- 
cal ground state with an excited state, three of them de- 
scribing an excitation and three a deexcitation, and only 
these six operators are physically relevant. Out of the 
three excitations (deexcitations) , two are transverse, i.e., 
change the spin, and one is longitudinal, i.e., does not 
affect the spin. For a classical phase with L sublattices 
one therefore has 4L transverse and 2L longitudinal op- 
erators per unit cell. Since the spin-orbital Hamiltonian 
(^.9|) does not couple transverse and longitudinal opera- 
tors, this yields also 4L transverse and 2L longitudinal 
modes. Because of time- reversal invariance they all occur 
in pairs with opposite frequencies, ±w-"''. 

k 

Finally, the SU (2) spin invariance of the Hamiltonian 
guarantees that the transverse operators raising the spin 
are decoupled from those lowering the spin, and that they 
are described by the same set of equations of motion, 
so that the transverse modes are pairwise degenerate. 
Such a simplification does not occur in the longitudi- 
nal sector. So, in conclusion, in an L-sublattice phase 
there are L doubly-degenerate positive-frequency trans- 
verse modes and L nondegenerate positive-frequency lon- 
gitudinal modes, accompanied by the same number of 
negative-frequency modes. This may be compared with 
the well-known situation in the HAF, where there is, with 
only spin operators involved, only one (not two) doubly- 
degenerate positive-frequency (transverse) mode in the 
two-sublattice Neel state. 
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For the actual evaluation it is convenient to decompose 
t he s uperexchange terms in the spin-orbital Hamiltonian 

(PI), 



n.j = n\\+n^, (4.7) 

into two parts which depend on the bond direction: 

(i) for the bonds {ij) \\ (a, 6), 

m\ 

■{SSjxx + S.JZZ + Xtj^/iKj) - 2riSi ■ Sj 
+ (1 + 2r7)(n,_ + A,jV3r,)(n,_ + AyVST,) 
-(3 + ry)], (4.8) 

where ~ (—1)*^ with y being a unit vector in the 
^-direction, and 

(ii) for the bonds (ij) _L (a, 6), i.e., along the c-axis, 

'H± = J [(4 - '2.ri)Sizz ■ Sjzz - -niSixx ■ Sjzz 

+ S^,z ■ Sjxx) + (1 + 277)n,_n,_ - i(3 + r;)] . (4.9) 

Here and in the following Sections we consider a 3D 
model with (3 — 1. We note that the orbital interac- 
tions (2.12) are quite different in i/y and H±; propagat- 
ing spin-and-orbital excitations are possible only within 
the (a, b) planes, where they are coupled to the spin exci- 
tations, while in the c-dircction only pure spin excitations 
and pure orbital excitations occur, which are decoupled 
from one another. This breaking of symmetry between 
and H± is a consequence of the choice of basis as \x) 
and \z) orbit als. 

In the following Sections we consider transverse and 
longitudinal excitations in the various symmetry-broken 
states. The transverse excitations, i.e., spin- waves and 
spin-and-orbital-waves, are calculated using the spin- 
changing operators which make a transition to a state 
realized in a classical phase at a given site i; for example 
for the AFxx phase these operators are for i in the A 
(spin-up) sublattice. 



Kt. 



■ Ax]'^izl- 



(4.10) 



The longitudinal excitations without spin-flip are most 
conveniently obtained starting from spin-dependent or- 
bital excitation operators. 



T,, 



d} d- 



(4.11) 



The commutation relations for these operators are pre- 
sented in Appendix O. 



B. Antiferromagnetic AFxx phase 

The nature and dispersion of elementary excitations in 
the spin-orbital model (|2.9D can be conveniently studied 



in the leading order of the 1 /S expansion using the Green 
function formalism. We note, however, that equivalent 
results for the AFxx and AFzz phases can be obtained us- 
ing instead an expansioft, around a classical saddle point 
with Schwinger bosons.Ell 

We start from the equations of motion for the Green 
functions generated by the excitatioa-Qperators ( 4.1C ) 
written in the energy representationJ2ai£j 

E{{Stxx\-)) - ^{[SL^-]) + {{[SL^H]\-)). (4.12) 
E{{KL\-)) = ^{[KL^ ■•■]> + mL,H]\...)), (4.13) 



where the average of the commutator on the right hand 
side, e.g. {[S^x' ^j'xx])^ evaluated in the classical 
ground state. The excitation operators were chosen as 
leading to the local states \ix t) realized at one of the 
sublattices in the ground state of the A Fxx phase . As 
usually, the commutators in Eqs. ( 4.12 ) and ( 4.13 ) gen- 
erate higher-order Green functions. In contrast to the 
HAF, it does not suffice to consider the spin-flip Green 
function {{S^^^\...)), as the spin-flips may also occur to- 
gether with an accompanying orbital-flip, as described by 

We derived the equations of motion for the 
Green functions generated by the set of operators 
{Stxx:Ktxz,S!^xx^Kjxz}^ '"'liere i e A and j e B, and 
used the random-phase approximation (RPA) for spin- 
like operators which lineari|Kes-|thc equations of motion 
by a decoupling procedure.E§E^ Thereby, the operators 
which have nonzero expectation values in the considered 
classical state give finite contributions, e.g. for the first 
spin-fiip Green function one uses 



ii^ixx^mxxl' ")) i^nixx} {{^ixx\ ^ 



(4.14) 



and a similar formula for the mixed spin-and-orbital ex- 
citation described by {{K^J...)), 



{{E^txz^mxx\---)) — {^mxx){{^txz\---)) ■ 



(4.15) 



It is crucial that the decoupled operators have different 
site indices, and thus the decoupling procedure preserves 
the local commutation rules given in Appendix |b[ In- 
stead, if one uses products of spin and orbital operators, 
e.g., Kf^^ = S'+p^+ one is tempted to decouple these 
operators locallyEjO which would violate the algebraic 
structure of the so(4) Lie algebra. 

In the present case of the AFxx phase one uses the 
respective Neel state average values. 



i^jxx 



_ 1 
jxx/ 2 ' 

^i-) ~ 2' 



(4.16) 
(4.17) 



where i E A and j G B, and A and B are the two sublat- 
tices in a 2D lattice for the AFxx phase. All the remain- 
ing averages vanish, as this phase has a pure |a;)-orbital 
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character at every site, which simphfies significantly the 
equations of motion which resuh from the RPA proce- 
dure. 

The translational invariance of the Neel state impUes 
that the transformed Green functions are diagonal in the 
reduced Brillouin zone (BZ). As in the HAF, the Fourier 
transformed functions are defined for the Green functions 
which describe the spin dynamics on a given sublattice, 
either A or B. For instance, the pure spin-flip Green 
functions arc transformed as follows, 



■))a = 



•»i 



1 



+ I 

IXX I 



1 



■»A, 

■■))b, 



(4.18) 



where N is the number of sites in one sublattice. Hence, 
the problem of finding the elementary excitations of the 
considered spin-orbital model ( ^.9D reduces to the diag- 
onalization of a 4 x 4 dynamical matrix at each fc-point, 
as given in Appendix 

The symmetric positive and negative eigenvalues 
±07^"', with n — 1,2, solved from the matrix in Eq. ( |C^ ) 
may be written in the following form for the AFxx phase, 

J2 



\2 I ^2 



± j 

- 4(A 



zk k 
\2 2 



y^xk k xk' 



1/2 



(4.19) 



Here the quantities Aq and Tq play the role of local po- 
tentials and follow from the model parameters, Ez and 



Xr, 



(4.20) 
(4.21) 



= - 4:1] - 2 - 1] + Ez- 

The remaining terms are /c-dependent, and depend on 



7+(fc) = i(cos kx + cos ky), 
7_ (fc) i(cos kx — cos ky), 
^z{k) = cosfc^. 



(4.22) 
(4.23) 
(4.24) 



The quantities and P_^g for the AFxx phase take the 
form. 



(|-377)7+(A:), 



P^, = ix/3(3-77)7_(fc), 



xk 



while the last dispersive term, 



(4.25) 
(4.26) 

(4.27) 



carries no index and remains identical for both AF phases 
(AFxx and AFzz). We emphasize that the coupling be- 
tween the spin-wave and spin-and-orbital-wave excita- 
tions occurs due to the terms cx P^j:, as seen from Eq. 
(|C2D. It vanishes in the planes of k^ = rLky, but other- 
wise plays an important role, as discussed in Sec. 0. In 
the limit of large ^ oo, Eq. ( |4.19| ) reproduces the 
spin-wave excitations for a 2D antiferroniagnet with an 



AF superexchange interaction of J(^ 



k 



(4.28) 



while the dispersion of the high-energy spin-and-orbital 

(2) 

excitation, ujl'~Ez, becomes negligible. As explained 
above, both modes are doubly degenerate. 

Consider now the orbital (excito nic) excitations gener- 
ated by the orbital-flip operators (4.11). They are found 
by considering the equations of motion. 



.]) + mam,H]\...)), 

(4.29) 

.]) + m^0^,H]\...)), 

(4.30) 



E{{T^aPT ]■■■)) = -^{[TiaPh 



Ztt 



and the commutators are calculated using the rules (|B7|). 
In general, one finds four different excitation operators at 
each site. However, making a Four ier transformations as 
for the transverse operators ( 4.18 ), one may show that 
only two operators per sublattice suffice to describe the 
modes in an antiferromagnet. The structure of the re- 
spective RPA dynamical matrix is given in App endi x |^. 
The orbital excitations which follow from Eq. ( |C3| ) are 
in general given by 



Ck^ J ha(ua ±2p^j:)] 



1/2 



(4.31) 



yielding two, in general nondegenerate, positive- 
frequency modes. In the AFxx phase one finds. 



Pxk 



'z - 377, 



(4.32) 
(4.33) 



It is important to realize that the propagation of longi- 
tudinal excitations, being equivalent to a finite dispersion 
of longitudinal modes, becomes possible only at 77 > 0. 
This follows from the multiplet structure of the excited 
states, which allows a spin-fiip between the orbitals in 
the \^Eg) and in the 5^ = component of the pA2)-state 
only if Jh 9^ 0, as illustrated in Fig. ^. The processes 
~ txz are not shown, as they would lead to a final state 
shown in Fig. ||(b), i.e., to a propagation of a spin-and- 
orbital excitation which was already considered above. 
In contrast, the relevant longitudinal orbital excitation 
in the symmetry-broken state implies that the exciton 
has the same spin as imposed by the Neel state of the 
background; this state is shown in Fig. |8|(c). Therefore, 
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in a perfect Neel state without FM interactions due to 
77 7^ 0, only local orbital excitations are possible. These 
local excitations cost no energy in the limit of — ^ 
which demonstrates again the frustration of magnetic in- 
teractions at the classical degeneracy point, — -q = 0. 

An example of the excitation spectra is shown in Fig. ^ 
for the main directions in the 2D BZ, with X = (vr, 0) and 
S = (7r/2,7r/2). Near the F point one finds a (doubly- 



k at 



degenerate) Goldstone mode uj^^^ with dispersion 
fc ^ 0, as in the HAF, and a second (doubly-degenerate) 



transverse mode at higher energy, uj^ 



(2) 



Wo + ak^ . Near 

F the Goldstone mode is essentially purely spin- wave, the 
second mode purely spin-and-orbital wave. With increas- 
ing k these modes start to mix due to the P^j: term along 
the r — X direction. This is best illustrated by the in- 
tensity measured in the neutron scattering experiments, 
which see only the spin-wave component in each trans- 
verse mode, as explained in more detail in Appendix 
The intensity xi'l) moves from one mode to the other 
along the T ~ X direction in the 2D BZ (Fig. H), demon- 
strating that indeed the lowest (highest) mode is predom- 
inantly spin-wave-like (spin-and-orbital-wave-like) before 
the anticrossing point, while this is reversed after the an- 
ticrossing of the two modes. Thus, we make here a spe- 
cific prediction that two spin-wave-like modes could be 
measurable in certain parts of the 2D BZ, in particular in 
the vicinity of an anticrossing, if only an AFxx phase was 
realized for parameters not too distant from the classi- 
cal degeneracy point. Unfortunately, for the realistic pa- 
rameters for the cuprates,c3 one finds Ez/J ~ 10 which 
makes the spin-and-orbital excitation and the changes of 
the spin-wave dispersion hardly visible in neutron spec- 
troscopy. 

The orbital (longitudinal) excitations are found for the 
parameters of Fig. |^ at a finite energy, being of the same 
order of magnitude as the energy of the spin-and-orbital 

f2) 

excitation, oj- . The weak dispersion of these modes fol- 

k 

lows from the spin-flip processes in the excited states, as 
explained in Fig. || and discussed above. We emphasize 
that the orbital mode has a gap and does not couple to 
any spin excitation. At the classical degeneracy point M 
the orbital mode falls to zero energy and is dispersion- 
less, expressing that the orbital can be changed locally 
without any cost in energy. 



C. Antiferromagnetic AFzz phase 

The transverse excitations in the AFzz phase are de- 
termined by considering the co mplem entar y set of Green 
functions to that given in Eqs. ( |4.12| ) and ( |4.13| ), 



E{{SLl--)) = 2^{iSL,---]) + {{[SL.H]\-)), (4.34) 
E{{K+J...)) = ^(fe, ...]) + m+^,H]\...)), (4.35) 



with the excitations to the local \iz t) states. As usu- 
ally, the average of the commutator on the right hand 
side is next evaluated in the classical ground state. After 
obtaining the RPA equations, we thus use the following 
nonvanishing averages. 



= = 



(4.36) 
(4.37) 



in the AFzz phase. This leads again to the general form 



with all the elements except for i?j replaced by, 

A. = ' 



1-77 + 2(2-77), (4.38) 

= -i-77 + 2(l-27;)-e„ (4.39) 

Qzk = (5 - Vh+{k) + 2(2 - 77)7, (^), (4.40) 

n*; = 5^(i-^)7-(fc)- (4.41) 

Thus, the transverse excitations have the same form 



( 4.19 ) as in the AFxx phase, but the above quantities 
([4.38|)-(|4.41|) have to be used. 

In the limit of large i?^ — > — 00 one finds the spin- 
wave for a 3D anisotropic antiferromagnet with strong 
superexchange equal to 2 J(2 — 77) along the c-axis, and 
weak superexchange jJ{l — 277) within the (a, 5)-planes, 



J'^ = J 



{ [(1-77) + 2(2 -77)]' 
(5-^)7+(fc) + 2(2-77)7, 



1/2 



(4.42) 



while the spin-and-orbital excitation, lu'- 

k 



(2) 



Ez, is dis- 

persionless. Again, both these transverse modes are dou- 
bly degenerate. The orbital excitations in the AFzz phase 
are f ound using the equations o f mo tion of the form ( 4.29| ) 
and (|4.3G|) which lead to Eq. (|4.3l|) with. 



^z - 3ri, 



(4.43) 
(4.44) 



and we find again zero-energy nondispersive modes at 
= 77 = 0. 

The representative excitation spectrum for the AFzz 
phase is shown in Fig. |l^. We use the 3D BZ for a 
bcc lattice with the standard notation: W — (tt, 7r/2,0), 
L = (7r/2, 7r/2, 7r/2) and K = (37r/4, 37r/4, 0). The trans- 
verse modes have qualitatively the same behavior as in 
the 2D AFxx phase, and one finds a Goldstone mode w-^"* 

k 

at the F point which is spin-wave-like, accompanied by 

(2) 

a finite energy spin-and-orbital mode . The first one 
is linear, while the second changes quadratically with in- 
creasing k. The dispersion in the T — X direction is, how- 
ever, only ~ 0.7J, while in the AFxx phase a large dis- 
persion of ~ 2.5 J was found (Fig. ^). This demonstrates 
the very large difference between the superexchange in 
the (a, 6)-planes in the two AF phases. 
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Here one should bear in mind, that in a strongly 
anisotropic antiferromagnet, such as the AFzz phase, 
the dispersion of the spin-wave mode in the {kx,ky) 
plane is roughly (2 J^b Jc)^^^, so actually enhanced by 
{Jc/'2.JabY^'^ compared with the planar exchange con- 
stant. In fact, there is also strong mixing between spin 
wave and spin-and-orbital wave along F — X, depress- 
ing at the X-point w^'' by no less than 0.5J from its 
pure spin- wave value. The mixing effect is also visible in 
the relatively large neutron intensity of the second mode. 
By contrast, the transverse excitations are rather pure all 
along the W — L direction [where the neutron intensity 



x{q) is larger], except in the regime where and 
the neutron intensity is distributed between the modes. 
However, owing to the abruptness of the anticrossing, the 
range where the modes have simultaneously appreciable 
intensity is very narrow, and their energetic proximity 
then makes it likely that they would be measured as a 
single broad maximum. 

The (longitudinal) orbital excitation is found at the X 
and L points at the same energy as that of a local excita- 
tion from \z) to \x) orbital (see Fig. |l^). It depends only 
on the energy difference between the orbitals, and has 
a weak dispersion by the same mechanism as described 
above for the AFxx phase (Fig. ||). 



D. Mixed-orbital FFA phase 



The excitation operators which couple to the local states in a sym metry-broken phase with mixed orbitals are 
linear combinations of the operators considered in Sees. [VB| and IV C. It is therefore convenient to make a unitary 
transformation of the Hamiltonian (^|^) to new orbitals defined as follows for i G A or i e sublattice. 



cos W 
— sin( 



smt 
cos( 



and for j e B or j e C sublattice, 



{ \JIA ^ _ 



cos W — sm ( 
sin 6 cos 9 



\iz) 
\ix) 



\3z) 



(4.45) 



(4.46) 



With these definitions and by choosing the angle Q at the value which minimizes the classical energy (3.11), we 
guarantee that and respectively, are at each site the orbital state realized in the classical MOffa phase, 
which is G-type with respect to the orbital ordering, while \iv) and \jv) are the excited state, so that one can readily 
define the excitation operators pertinent to the symmetry-broken ground state of this phase. Thus the spin, spin- 
and-orbital, and orbital operators in terms of the new orbital states defined by Eqs. (4.45) and (4.4(;) 
are 



I), 

f \iva){iiia\), 



Mi- = i ^(|i/icr) («^cr| — liiya) {iiy<7\) . 



(4.47) 
(4.48) 
(4.49) 

(4.50) 



The new operators: JCtafS, % and Mi- fulfill the same commutation rules as the nontransformed operators: Kiaf3, 
Ti , and ni_ , respectively; they are given in Appendix To simplify the notation we also introduce total spin and 
spin-and-orbital operators. 



Si — Si^^j, 



S'luf , 
^iua 



(4.51) 
(4.52) 



The Hamiltonian ( |2.9| ) has to be transformed by the inverse transformations to those given by Eqs. ( 4.45| ) and 
(4.46). For the bonds {ij) || (a, b) with i E A and j E B one finds. 



i J XI { (1 ~ 5^?) [((2 - cos 26')4.M + (2 + cos 20)S^^^ + sin 26*^,) 
X ((2 - cos 20)Sj^^ + (2 + cos 29)Sj^^ - sin 29IC^) 
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+3{sm29{Sif^^ - + cos26i/Ci)(sin26'(5j^^ - Sj^,y) + cos29JCj) 
+ X^jVi (((2 - COS 261)51^^ + (2 + cos29)S^^^ + sin 26'£,) (sin 261(5^^^-4^^) " cos26'£j 

-(sin26'(5',p^ - Si^^) + cos2eK.,){{2 - cos20)Sj^,f, + (2 + cos2e)Sj^^ - sm29ICj) 

{cos29{Sif^^ - Si^^) - sin26'/Ci)(cos26'(5j^^ - Sj^^) + sin20ICj) 
-3(sin26'(5.,p^ - S^^^,) + coa20K.i){sm20{Sj^^ - Sju„) - cos20ICj) 



-XijVi (^{cos20{Sifj_f_, - Si^i,) - sin20ICi){siii20{Sj^^ - Sj^^) - cos29ICj) 
+ (sin26'(5j^^ - Si^^) + cos26'/Cj)(cos26'(5j^^ - 5^^^) - sm29ICj' 
+ (1 + 27/) (cos 29N'^ - sin 26*7^) (cos 26'A/)- + sin 29fj) 
-3(sin26'M + cos 26''7^)(sin 26'A/) - cos29%) 
-AyV3 (^(cos2e'M - sin 26*7; ) (sin 26'A/) - cos29fj) 

+ (sin 29J\l'i + cos 29fi){cos 29Mj + sin 207^) j - (3 + ry) | , 
while for the bonds {ij) _L (a, 6) it takes the form 

H^ = J^[{1- ir7)((l + cos2e')5,^^ + (1 - cos 2e')5,^^ - sin 26*^,) 

X ((1 + COS 29)Sjf,f, + (1 - cos 26l)4i/i. - sin 29ICj) 
- ((1 - cos 29)S^f,^, + (1 + cos 29)S,^^ + sin 26£,) 

X ((1 + cos 29)Sjf,f, + (1 - cos 26')4^/,. - sin 29ICj) 
+ ((1 + cos 261)5,^^ + (1 - cos 261)5;^^ - sin 29ICi) 

X ((1 - cos 26l)4>M + (1 + cos 26')4^,. + sin 26'^j) 

+ (1 + 2r]){cos29Ati - sai29fi){cos29Mj - s\n29fj) - i(3 + 77)| . 

Finally, the transformed orbital-anisotropy term reads 

Hr = ^(cos26'M - sin 26*7;). 



2t]SiSj 



(4.53) 



(4.54) 



(4.55) 



The transverse excitations may be found starting from 
the relevant raising operators that lead to the local 
state t) realized in one of the sublattic es, a nalo- 
gous to those introduced for the AFxx phase ( [4.10[ ), i.e., 
the set {Si^^,JC^^j^,Sj^^,ICj^^,Si^^ ,ICi.^j^,Si ,JCi }, 
where i G A, j G B, k G C, and I G D; they lead 
as usual to the orbitals {|«^), Ijm)} (0) realized in the 
MOffa phase. 



mst,\...)) = ^iK,,---]) + {{K,,H]\...)), (4.56) 



We applied the same RPA procedure as in Sees. [V B and 



IV C in order to determine the Green function equations 
in the fc-space. The longitudinal excitations can be ob- 



tained from opera tors s imilar to those used in the AFxx 
and AFzz phases ( 4.11 ), 



(4.58) 



for the (a, b) planes with the |-spins, and the corre- 
sponding Tifj,i,i and Tiu^i for the (a, b) planes with the 
|-spins. The commutation operators for these opera- 
tors are analogous to those presented in Appendix ^ 
and may be easily obtained. The resulting dynamical 
matrices for both transverse and longitudinal excitations 
are given in Appendix |c| their numerical diagonaliza- 
tion gave the modes presented below. There are four 
doubly-degenerate positive-frequency transverse modes, 
and four non-degenerate positive-frequency longitudinal 
modes, consistent with the MOffa phase having four 
sublattices. 
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An example of the transverse and lon gitu dinal modes 
in the MOffa phase is presented in Fig. |ll|. The modes 
are shown in the respective BZ which corresponds to the 
magnetic unit cell of the MOffa phase: The 2D part 
along F — X — iS* — Fis identical with the AFxx phase 
(compare Fig. H), reflecting the orbital alternation, while 
the AF coupling along the c-axis results in the folding of 
the zone along the T — Z direction, with Z' = (0, 0, 7r/2) 
and S' = (7r/2, 7r/2, n/2). One finds one Goldstone mode, 
and three other finite-energy modes at the F point. If no 
AF coupling along the c-axis is present, similar positive- 
energy modes describe the excitation spectrum in the 
MOff phase in the 2D part of the BZ (in the region of 
stability shown in Fig. |^), and the symmetric negative- 
frequency modes carry then no weight. In contrast, due 
to the strong AF interactions in the MOffa phase, the 
negative modes give a large energy renormalization due 
to quantum fluctuations, as discussed in more detail in 
Sec. 0. 

The spin-wave and spin-and-orbital-wave excitations 
are well separated along the F — X — S* — F path, with 
a gap of ~ 0.5 J, as the FM interactions (x Jrj are con- 
siderably weaker than the orbital interactions which are 
cx J. Therefore, the neutron intensity xio) is found 
mainly as originating from the lowest energy mode, \ 
with a small admixture of the higher-energy spin-and- 
orbital excitation, oj- 

k 



(3) 



The magnetic interactions are 
considerably stronger along the c-axis; the modes mix 
and the higher-energy excitations, wl"'' with n = 3,4, 
have a larger dispersion in the remaining directions with 
fcx 7^ 0. Strong mixing of the modes in this part of the 
BZ is also visible in the intensity distribution, with the 
modes n = 1 and n = 3 contributing with comparable 
intensities (Fig. |ll|). The fact that modes labelled as 2 
and 4 have zero intensity is due to the path F — Z' — S" — F 
being in the high-symmetry BZ plane where kx — ky so 

that 7-(fc) = 0. Then modes 2 and 4 have equal ampli- 
tude but are exactly out-of-phase between A and B sites 
as well as between C and D sites, and so their neutron in- 
tensities vanish, and only the companion in-phase modes 
1 and 3 are observable by neutrons. Unfortunately, no 
experimental verification of these spectra is possible at 
present, as the spin excitations measured in neutron scat- 
tering for KCuFa are consistent with the Bethe ansatz 
and thus suggest a spin-liquid ground state with strong 
ID AF correlations instead of the A-AF phase with mag- 
netic LR0.E3 

Interestingly, although the order in the (a, b) planes is 
FM, the energy of the Goldstone mode increases linearly 
in all three directions with increasing A:, and the slopes 
are proportional to the respective exchange interactions. 
This behavior is a manifestation of the A-AF spin order; 
a qualitatively similar spectrum is found experimentally 
in LaMn03,O where, however, the excitation spectra de- 
scribe large spins S" = 2 of Mn'^"'' ions. The rather small 
dispersion of the spin-wave part at low energies is due 



to small values of the exchange constants for the actual 
optimal orientation of orbitals found at Jh/U — 0.3. We 
note, however, that the AF interactions along the c-axis 
are much stronger at Jh — *■ than in the present case. 
The AF structure along the c-axis may be easily recog- 
nized from the symmetric spin-wave mode in the T — Z 
direction with respect to Z' = (0, 0, 7r/2), while this mode 
increases all the way from the F to the X point. The fact 
that only two modes have nonzero neutron scattering in- 
tensity along F — Z' — S" — F is due to this BZ path be- 
ing in the high-symmetry BZ plane, where kx = ky and 
7_ (fc) = 0. Then two modes have equal amplitude but 
are exactly out-of-phase between A and B sites as well 
as between C and D sites, and so their neutron intensi- 
ties vanish, while only the companion in-phase modes are 
visible to neutrons. Unlike in the AF phases, the purely 
orbital excitation is here energetically separated from the 
spin-wave and spin-and-orbital-wave modes. The disper- 
sion is quite small and decreases with rj. 



E. Mixed-orbital AFF phase 

The elementary excitations in the MOaff phase may 
be obtained using a similar scheme to that used in Sec. 
IV D for the MOffa phase. First of all, one defines new 



quantum states which correspond to the minimum of the 
classical problem. This is realized by a unitary transfor- 
mation of the Hamiltonian to the new orbitals defined 
for i G A sublattice as, 



\iv+) 



cos sin 9+ 
— sin 6*+ cos 6+ 



and for j £ B sublattice as, 



cos f _ — sm ( 
sin 6'_ cos 6- 



\iz) 
\ix) 



(4.59) 



(4.60) 



By choosing the angles Qj^ and Q- at the values which 
minim ize the classical energy, given by Eqs. ( [3.14 ) and 
(|3.14 ), we guarantee that and Ij/J.-), respectively, 

are at each site the orbital state realized in the classical 
MOffa phase, and one may easily define the new exci- 
tation operators with respect to the symmetry-breaking 
which occurs in this phase; they are analogous to those 
given in Eqs. ( 4.47] )-(4.52). Next, the Hamiltonian is 
rotated to the new representation as described in Sec. 
IV D . We do not present an explicit form of the spin- 
orbital Hamiltonia n (|2.9| ) in th is case, as it may be ob- 
tained from Eqs. ( [4.53D -( 4.55 ) by replacing the angle d 
by and Q- for the sublattice A and B, respectively. 
Furthermore, due to the degeneracy between the MOaff 
and MOfaf phases, we had to average the crystal-field 
between the two sublattices in the actual calculation. 

We have verified that the transverse excitations have 
a similar dependence on the fc-vector to those found in 
the MOffa phase, and we show the representative data 
in Fig. O. For convenience, we have rotated the BZ and 
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use just the same notation as in Fig. |TT|. The value of the 
crystal-field Ez is in the present case effectively smaller 
by a factor of two in comparison with the MOffa phase. 
This asymmetry is a consequence of the choice of \x) and 
\z) states as the orbital basis. 

One finds again that the spin-wave and spin-and- 
orbital-wave excitations are well separated along the 
T — X — S — T path, and the gap between them has 
increased to ~ 1.2 J. We note a stronger renormalization 
of the low-energy modes which follows from weakened 
FM interactions between the alternating orbitals in the 
(&, c)-planes in the present case as compared with those 
within the (a,5)-planes in the MOffa phase. Although 
the orbital excitations are still well separated from the re- 
maining transverse modes, their dispersion is larger than 
that in Fig. [l[ 



V. QUANTUM FLUCTUATIONS 

The size of quantum fluctuation corrections to the clas- 
sical order parameters determines the stability of the 
classical phases. As mentioned in Sec. |, frustration of 
magnetic interactions leads in spin models to divergent 
quantum corrections within the LSW theory. Before cal- 
culating these corrections in the present situation, a gen- 
eralization of the usual RPA procedure to a system with 
several excitations is necessary. Here we present only the 
relations needed to calculate the quantum corrections to 
the LRO parameter and ground state energy, while more 
details will be reported separatelyO 

For that purpose, let us denote here the local operators 
constituting the so(4) Lie algebra at site i as Hubbard 



operators, Xf^ — \ia){il3\. Using the unity operator, 
X^^A'f'^ = 1, the diagonal operator that refers to the 
state \ia) realized at site i in the classical ground state of 
the phase under consideration may be expanded in terms 
of the excitation operators. 



(5.1) 



while the diagonal operators referring to an excited state 
|i/3) are expressed as 



X. 



/3/3 



f3a -^af3 



xrx. 



(5.2) 



Applying these equations to the z-th spin component 
5,f — Sf^^ + S^zz of total spin at site i in one of 
the AF phases with pure orbital character (say AFxx for 
definiteness), one finds, for i in the spin-up sublattice,0 



x'\ ,x'\ 



X 



X. 



X 



IZX IXZ 



(5.3) 



Taking the average one obtains, with the MF value 

i^i) = 2' 



{St 



I RPA 



txx ixx / 



2 (Sixx^i 

= {St) - s{st] 



{K-Kt) 



(5.4) 



where the second equality is valid because averages like 
(SfxxSi'zz) since they involve 'ghost' modes, so 

that one may formally replace by S^^^^ + Sf^z = Sf , 
etcetera. The first contribution esc {S~ S^) is the usual 
renormalization due to spin waves, while the second term 
cx {K~ Kf) stands for the reduction of (S*!" )rpa due to 
spin- and-orbital- wave excitations. Both terms involve a 
local excitation preceded by a deexcitation which repro- 
duces the initial local state. As expected only the trans- 
verse excitations contr ibu te to the spin renormalization. 
Note that, since Eq. (5.3) is an exact operator relation, 
the present procedure gurantees that Eq. (5.4) is a con- 
serving approximation which respects the sum rule for 
the occupancies of all states, '^^pi^i^) = 1- The gener- 
alizat ion o f Eq. (5.4 ) to the MO phases using the opera- 
tors (4.47), (4.48), or to other order parameters, like the 
orbital polarization, is obvious. 

The local correlation functions which renormalize the 



order parpneter in Eq. (5.1) are determined in the stan- 
dard wayj23 



N ^ 



+ 00 



1 



exp(/?[x)) 



-J, (5.5) 



where f3 — l/fc^T, and 

AAB^iKco < 0) = 21in{(Ar\Bi)), 



= E-^ABt(fc)'^(^-4'^) (5-6) 

is the respective spectral density for the negative frequen- 
cies (v < 0), and ^^^^(fc) are the respective spectral 
weights. Therefore, the correlation functions at T = 
are found by summing up the total spectral weight at 
the negative frequencies. 



{BJA 



1 

N 



(5.7) 



As we show else where, O the Hamiltonian of the spin- 



orbital model (2^) may be expanded in RPA in terms of 
the excitation and deexcitation operators, 

H ~ TiuF + "Hrpa, (5.8) 
where TCmf is given by Eq. ( |3.l| ), and 

hrpa -EE xraT'xr' + E E ^f^'^''^' 



+ E E (^r^^"" + ^r^&^^f 



E E (^r^c^r + ^r"c^f ; 



(5.9) 
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for a two-sublattice phase (the generalization to the four- 
sublattice MO phases is straightforward). The MF part 
describes the classical pr oble m which was discussed in 
Sec. III. The RPA part (5.9) describes the many-body 



WAFxx(fc) ^Ax + Bx {kt + 14fc?fc^ + k: 



.1/2 



(5.12) 



problem in a linear approximation, with the fixed indices 
a and P referring to the symmetry-broken state at site i 
and j, respectively. This expansion leads, after changing 
the order of excitation operators X"^ to normal order, 
and after making straightforward transformations, to a 
compact expression for the average energy contribution 
per site, 



Erpa — -^(^rpa) 



k 



(5.10) 



where A is the matrix of positive on-site coefficients a'^ , 
a'^ , appearing in the first line of Eq. (^^), and with 
the sum running over all modes with positive frequencies 
(counting doubly-degenerate modes twice) in the reduced 
BZ. This expression is seen to be a direct generalization 
of the familiar result for the HAF, the distinction being 
that more modes contribute here, and so Eq. (5.10) rep- 
resents the energy gain (i?RPA < 0) due to the reduction 
in zero-point energy of the propagating modes in com- 
parison with that of the local excitations. We use Eq. 
(5.10) to calculate the total energy in RPA, 



E 



MF 



(5.11) 



Before discussing the renormalization of the order pa- 
rameter and the corresponding energies in RPA, we con- 
centrate ourselves on the behavior of the transverse ex- 
citations when the crossover lines between the classical 
phases are approached. As already emphasized in Sec. 
[V, the spin- wave and spin-and-orbital-wave excitations 
couple. As a consequence, the modes in all considered 
phases soften when the transition lines between differ- 
ent classical phases, or classical degeneracy point are ap- 
proached. This softening is shown for a representative 
value of Jh/U = 0.3 in Fig. [l^ for the two AF phases. In 
the AFxx phase the energy scales of both excitations are 
separated for > 4:J, while the spin-and-orbital mode 
moves towards zero energy with decreasing E^, and fi- 
nally becomes soft at the X point, along k = (7r,0, fc^) 
and along equivalent lines in the BZ for E^ — 1.54J. A 
similar mode softening is found for the AFzz phase at 
Ez < 0, with the soft mode along T — X and equivalent 
directions in the BZ at E^ — —1.84 J. This pecular soft- 
ening along lines and not at points in the BZ shows that 
the modes behave 2D-like instead of 3D-like: constant- 
frequency surfaces are cilinders contracting towards lines, 
not spheres contracting towards a point. 

By making an expansion of Eq. (4.19) around the soft- 
mode lines, one finds that the (positive) excitation ener- 
gies are characterized by finite masses in the perpendic- 
ular directions: 



independently of k^ (here k^ ~ k^ — tt), and 

^AFzz(fc) A, + (fc2 + Akl) , (5.13) 

independently of kx-, and similarly along the F — y di- 
rection with ky replaced by kx- As an example we give 
explicit expressions for the AFxx phase at ?7 = 0, 
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2e, + 3' 



Bx — 
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(5.14) 



where one finds that the gap A^; — > when Sz — > 0, i.e. 
upon approaching the M = {Ez,Hh) = (0,0) point at 
which the AF order is changed to the AFzz phase. This 
illustrates a general principle: A^ — when the crossover 
line to another pha se is approached, and Bi ^ when the 
modes ( 5.12| ) and ( 5.13 ) soften, making quantum fluctu- 
ation corrections to the order parameter to diverge log- 
arithmically, (SS) - / d?k/Lo{k) ~ / (Pkl{Ai + Bik^) ~ 
InAi. We emphasize that for the occurrence of this di- 
vergence not only the finiteness of the mass but also the 
2D-like nature of the dispersion is essential. It enables 
a 3D system to destabilize LRO by what are essentially 
2D fluctuations. So the divergence of the order param- 
eter near the cross-over lines in the phase diagram and 
the associated instability of the classical phases, may be 
regarded as another manifestation of the effective reduc- 
tion of the dimensionality occurring in the spin-orbital 
model. We do not present explicitly the softening of the 
longitudinal modes which also happens at the transition 
lines. 

A seemingly attractive way to simplify the calculation 
of the transverse excitations would be to make a decou- 
pling of the spin-waves and spin-and-orbital- waves. How- 
ever, this is equivalent to violating the commutation rules 
between tiie spin and spin-and-orbital operators in Ap- 
pendix pltj and this changes t he ph ysics. It gives the 
same excitation energies as Eq. ( [4.19| ) , bu t with P^^: = 0; 
the numerical result is given in Fig. |lj. Of course, the 
spin-wave excitation does not depend then on the or- 
bital splitting Ez, and the spin-and-orbital-wave excita- 
tion gradually approaches the line uj^ — Q with decreasing 
\Ez\. It has a weak dispersion which depends on Jh and 
on the value of | , and gives an instability at the F point 
only, not at lines in the BZ, and in the phase diagram 
well beyond the transition lines of Fig. ^, i.e., within 
the MOffa and MOaff phase for Ez < and Ez > 0, 
respectively. Such spin-wave and spin-and-orbital-wave 
modes give, of course, much smaller quantum corrections 
of the order paran^eter and energy than the correct RPA 
spectra of Fig. 
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The spin-waves in the MOffa phase, stable at Ez < 0, 
soften with decreasing 77 (2.7), as shown in Fig. ^ At 
large 77 the spin-and-orbital-waves at high energies are 
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well separated from the spin-wave modes. The latter 
have a rather small dispersion at Jh/U = 0.3 which fol- 
lows from relatively weak FM interactions in the (a, h) 
planes, and AF interactions along the c-axis. The modes 
start to mix stronger with decreasing 77, and finally the 
gap in the spectrum closes below 77 — 0.1. The mode soft- 
ening occurs again along lines in the BZ, namely along 
the T — X direction. Unfortunately, we could not perform 
an analogous analytic expansion of the energies near the 
softening point to that in the AFxx and AFzz phases, 
but the numerical results reported here suggest a qualita- 
tively similar behavior to these two phases. The MOaff 
phase gives an analogous instability ai Ez > 0. 

The soft modes in the excitation spectra give a very 
strong re nor malization of the order parameter (S'^)rpa 
in RPA (5.4) near the mode softening, as shown in Fig. 
|l6| . The quantum corrections exceed the MF values of 
the order parameter in the AFxx and AFzz phases in 
a region which separates these two types of LRO. Al- 
though one might expect that another classical phase 
with mixed orbitals and FM planes sets in instead, and 
the actual instabilities where 5{Sz) — > 00 are found in- 
deed beyond the transition lines to another phase, the 
lines where 5{S^) = (S^) occur still before the phase 
boundaries in the phase diagram of Fig. (see Fig. 1 of 
Ref. ^ . This leaves a window where no classical order is 
stable in between the G-AF and A-AF spin structures. 

The origin of such a strong renormalization of (S^) 
may be better understood by decomposing the quantum 
corrections into individual contributions as given in Eq. 
(5.4) (see Table I). The leading correction comes from the 
local spin fluctuation expressed by {S~ S^) and enhanced 
with respect to the the pure spin model (HAF), while the 
spin-and-orbital fluctuation, {K~ K^), increases rapidly 
when the instability lines (5'^)rpa — are approached. 
Interestingly, the latter fluctuation is stronger in the 
AFxx than in the AFzz phase for the same values of 
Jh and \Ez\ which demonstrates that the AFzz phase is 
more robust due to the directionality of the \z) orbitals 
and the strong AF bonds along the c-axis. This asym- 
metry is also visible in Fig. |l6|, where (S'^)rpa decreases 
somewhat faster towards zero for Ez> Q- 

In both G-AF phases (AFxx and AFzz) the leading 
contribution to the renormalization of (iS'^)rpa comes 
from the lower-energy mode, especially at larger values 
of Jh- In the case of Jh = one finds, however, that 
the contribution from the lower mode either stays ap- 
proximately constant (in the AFxx phase), or even de- 
creases (in the AFzz phase) when the line of the col- 
lapsing LRO is approached at \Ez\ — > (Table I). This 
latter behavior shows again that the coupling between 
the spin-wave and seiji-and-orbital-wave excitations is of 
crucial importance.E^ This is further illustrated by Fig. 
|l7| , which shows the renormalization of {Sz) as obtained 
when spin waves and spin-and-orbital waves are decou- 
pled in the manner discussed above. One observes that 
significant reduction of {Sz) then sets in only very close 
to the actual divergence. 



Also the orbital polarization is renormalized by the 
quantum fluctuations, but this is a rather mild effect 
not showing any instability, since this renormalization 
involves only the spin-and-orbital and the orbital excita- 
tion but not the spin excitation, which is the one partic- 
ipating most strongly in the lowest transverse mode that 
goes soft. This is seen in Fig. 
the occupation of the \x) orbital, again for Jh/U 



where we show (.n-x), 
0.3, 

both at the MF level as well as including the RPA quan- 
tum fluctuations, calculated from an expression similar 



to Eq. (5.4), e.g. in the AFxx phase from 

= 1-4 {T,zxT^xz) - {KrK+) 



(5.15) 

Especially in the MOffa and MOaff phases the devia- 
tion from the classical valu e of 9 as given by Eq. ( 3.11 ) 
and by Eqs. (3.14) and (3.15), respectively, is small. 
Only in the AFxx phase a significant admixture of | z) oc- 
cupancy could occur close to the regime where this phase 
becomes unstable due to the divergence of (S'^)rpa. 

The reduction of (5'=)rpa in the MOffa/MOaff 
phases (Table II), described by a relation similar to Eq. 
( |3.4[ ), is in general weaker than that in the G-AF phases. 
This is understandable, as the quantum fluctuations con- 
tribute here only from a single AF direction, while the 
FM order in the planes does not allow for excitations 
which involve spin flips and stabilizes the LRO of A-AF 
type. For fixed Jh one finds increasing quantum correc- 
tions S{S^-) when the lines of phase transitions towards 
the AF phases are approached. These corrections in- 
crease faster with increasing \Ez\ in the MOffa phase, 
as the increasing occupancy of the 1 2) -orbital makes the 
AF interaction stronger there than in the MOaff phase, 
where the occupancy of the \x) orbital increases slower 
roughly by a factor of two. This qualitative difference 
between these two A-AF phases may be seen in Fig. |l^. 
As in the G-AF phases, we find that the two lower-energy 
modes give the larger contribution to the renormalization 
of the order parameter. The spin-and-orbital fluctuation 
{JC~ ICf) remains almost independent of Ez, but increases 
with decreasing values of Jh- Thus we conclude that the 
collapse of the LRO in the A-AF (MO) phases is primar- 
ily due to increasing spin fluctuations, (S^S^), while the 
spin-and-orbital fluctuations become of equal importance 
only when the multicritical point of the Kugel-Khomskii 
model M = {Ez, Jh) = (0, 0) is approached. 

The representative quantum corrections to the ground 
state energy are given in Table III. First of all, these 
corrections are larger by roughly a factor of two in the 
G-AF phases (AFxx and AFzz) than in the A-AF phases 
(MOffa and MOaff/MOfaf). We beheve that this is 
a generic difference between the quantum corrections in 
the A-type and G-type AF phases, with the latter sta- 
bilized more due to the spin fluctuations contributing 
at all the bonds. Therefore, the G-AF phases win over 
the A-AF ones near the transition lines, as for example 
found at Jh/U — 2.0 and Ez/J — 0.2. However, one 
should keep in mind that the energy alone does not suf- 
fice for the stability of a particular phase in RPA, since 
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the MF value of the order parameter, (S^), has to remain 
larger than the respective quantum correction, S{S^). 
Secondly, the 2D AFxx phase is characterized by larger 
quantum corrections than the strongly anisotropic AFzz 
phase at the same values of Jh/U and \Ez\/J. The same 
observation was made before at the multicritical point 
M = {Ez,Jh) — (0,0) S This is not surprising since 
the 2D HAF is already quite close to the disordered spin 
state. We note that the energy gain due to quantum fluc- 
tuations of 0.423J (obtained for the actual interactions 
of I J in a 2D HAF) is there considerably smaller than 
the values of 5E of the order of 0.65J reported in Table 
III. 

Finally, we note that the dominating contribution to 
the quantum corrections to the energy comes from the 
transverse excitations. The longitudinal excitations do 
not contribute at all at Jh/U = 0, where these modes are 
dispersionless. Otherwise, the orbital excitations have al- 
ways a significantly smaller dispersion than the value of 
the orbital gap in the spectrum, and the resulting quan- 
tum corrections are therefore almost negligible. 



VI. SUMMARY AND CONCLUSIONS 



Summarizing, we have presented here the case that a 
generic (Kugel-Khomskii) model for the dynamics of an 
orbitally degenerate MHI is characterized by a number 
of peculiar features. In this paper we have followed a 
semi-classical strategy. Assuming that the ground state 
exhibits some particular classical spin- and orbital or- 
der, the stability of this order can be investigated by 
considering the Gaussian fluctuations around this state. 
In this way we flnd that in various regimes of the zero- 
temperature phase-diagram, conventional order is de- 
feated by the quantum fluctuations, and we expect a 
qualitative phase diagram as shown in Fig. ^ 

In the first place, near the transition lines between 
the different phases modes soften, and these soft modes 
cause the zero-point fluctuations to diverge. This is not 
dissimilar from the general theme associated with the 
geometrically frustrated quantum spin-models, like the 
Ji — J2 — J3 model.c3 A significant difference is that in 
the present case the source of the problems is distinct: it 
is associated with the difficulty to simultaneously satisfy 
the requirements for a stable spin- and orbital order. The 
cause of the frustration is dynamical instead of geomet- 
rical. 

The most interesting feature is the point at the origin 
of the phase diagram. On the classical level it is a point 
in the zero-temperature phase diagram where a quasi-lD 
antiferromagnet (MOffa phase), a 2D antiferromagnet 
(AFxx phase) , and a mildly anisotropic 3D antiferromag- 
net (AFzz phase) become degenerate (Fig. In fact, 
these possibilities make up only an infinitisimal fraction 
of the total degeneracy characterizing this special point. 
In addition, the orbitals can be freely rotated on every 



site, if the spins form a 3D antiferromagnet. Likewise, 
the phase diagram of Fig. 19 is highly incomplete. Next 
to Ez , there exist an infinity of other axes emerging from 
this special point, all corresponding with distinct ways 
of explicit local symmetry breaking in the orbital sector. 
One can either call this point an infinite-critical point, or 
a point of perfect dynamical frustration, or a point where 
local symmetry is dynamically generated. 

The obvious problem is that the above wisdom ap- 
plies only when quantum-mechanics does not play a role. 
Physical reality is different, and since the classical limit is 
pathological, quantum-mechanics is bound to take over. 
Although we have not found a way to make the case pre- 
cise, it appears to us that the local symmetry referred 
to in the previous paragraph exists only in the classi- 
cal limit. For this to be active on the quantum level, it 
should be that the true ground state is also highly degen- 
erate. Although we did not prove the uniqueness of the 
quantum ground state, so much is clear that the classical 
local symmetry gets lifted at the moment that quantum 
fluctuations become significant: the cancellations occur 
only if the spins are fully classical. Regardless the nature 
of the true ground state, it is generated by a quantum 
order-out-of-disorder mechanism.c2l 

The first possibility is a straightforward order-out-of- 
disorder physics: the quantum fluctuations affect the en- 
ergies of the various classical states in different ways, 
thereby breaking the classical degeneracy. One of the 
saddle points might get uniquely favored and this is what 
is suggested in Ref. ^ where it was argued that the 
AFzz phase becomes the ground state at the origin of 
the phase diagram. Although this is a credible possibil- 
ity, one would have to demonstrate that the other possi- 
bilities ari&Jess favoured, and moreover, we have showed 
elsewherdE -that the actual calculation by KhaliuUin and 
OudovenkoEil is flawed. The case is still open. 

Yet another possibility is unconventional spin- and or- 
bital order which is in a sense dual to the orbital- and spin 
(anti)ferromagnetism characterizing the 'classical' order: 
spin-orbital (resonatiag) valence bond (R)VB states. We 
demonstrated befordj that these straightforward gener- 
alizations of the spin RVB states, well known from the 
study of quantum spin-problems, appear as exceptionally 
stable. In a negl publication we will further elaborate on 
these matters.CHi 

The status of both proposals is rather unsure: they 
rely at best on the variational principle and the true vac- 
uum can still be completely different. In this regard, 
some recent^xperiments on the system LiNi02 are quite 
interesting.^ In this material a Mott-insulator seems to 
be realized, characterized by a low spin {S = 1/2) eg de- 
generate Ni(III) state. One would naively expect this sys- 
tem to be unstable towards a collective Jahn- Teller dis- 
tortion, accompanied by spin ordering. This indeed hap- 
pens in the closely related system NaNi02, buLin LiNi02 
ordering phenomena are completely absent ,Lj a pecu- 
liarity pointed out long ago.ES Instead, some quantum- 
critical state appears to be present, characterized by 
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power-law behavior of physical quantities, carrying un- 
usual exponents. Pending the magnitude of the Li- 
mediated kinetic exchange (JLi)7 one can view this sys- 
tem as either disconnected triangular layers of Ni(III) 
ions (vanishing Jli), or as interpenetrating cubic lat- 
tices of these ions which are described by the Kugel- 
Khomskii Hamiltonian (large JLi)El Hence, the peculiar 
state seen in the experiments can either originate iiuaome 
phenomenon associated with the triangular layers,Lll but 
it could also be related to the matters discussed in this 
paper. 

It is easy to settle this issue experimentally. Compare 
NaNi02 and LiNi02; if the physics of the quantum dis- 
order in the latter has to do with the (111) layers, one 
would expect on general grounds that in order to stabilize 
an ordered state, the effective dimensionality has to be 
increased, of course assuming that the basics of the elec- 
tronic structure (such like covalency) do not change ap- 
preciably. Hence, in this layer scenario one would expect 
stronger layer-layer interactions in NaNi02 as compared 



to LiNi02, following the standard result of quantum field 
theory that fluctuations increase upon lowering dimen- 
sionality. This standard wisdom does not apply to the 
Kugel-Khomskii model, however. The fluctuations find 
their origin in a dynamical frustration, and this frustra- 
tion is only present in three space dimensions. Hence, if 
the disorder in LiNi02 is caused by the physics discussed 
in this paper, its quantum magnetism should be rather 
isotropic in 3D space, while NaNi02 should be more 2D. 
It is noticed that according to elementary quantum chem- 
istry Li ions should be more effective in mediating kinetic 
exchange than Na ions. 
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APPENDIX A: DERIVATION OF THE SPIN-ORBITAL MODEL 



The derivation of the effective interactions between two (P ions at sites i and j takes the simplest form for a bond 
(ij) oriented along the c-axis. In that case the only nonvanishing hopping element is that between the two \z) orbitals 
on the neighboring sites, and thus the orbital occupancies in the initial and final d^d^ states have to be identical 
(apart from a possible simultaneous and opposite spin fiip at both sites). The possible initial states are described by 
a direct product of the total spin state, either a triplet {S = 1) or a singlet (S* = 0), and the orbital configuration, 
which takes one of four possibilities: \xiXj), \xiZj), \ziXj), or \ziZj). Moreover, the effective interaction vanishes if 
the holes occupy the \xiXj) configuration. The total spin per two sites is conserved in the d^d^ dj'^d^ excitation 
process, and therefore the spin dependence of the resulting second order Hamiltonian can be expressed in terms of 
the projection operators on the total spin states: {j + Si ■ Sj) for the triplet, and {j — Si ■ Sj) for the singlet. 

The general form of the effective Hamiltonian may be derived from the formula which includes all possible virtual 
transitions to the excited d^d^^ configurations. 



i.j)PiaPj[3, 



(Al) 



n,af3 



where t stands for the z — z hopping along the c-axis, Qs(i,j) is the projection operator on the total spin state, and 
Pia is the projection operator on the orbital state a at site i, while e„ stands for the excitation energies given by Eqs. 
( ^.5| ). The orbital projection operators on \x) and \z) orbital in the initial and final state of the d^ configuration at 
site i are, respectively. 



Pix = \ix){ix\ 
P%z = \iz){iz\ - 



(A2) 



where is defined as in Eqs. ( |2.12D . 

Therefore, one finds from Eq. (|A1|) for a bond [ij) along the c-direction. 



H. 
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£(3^2) 



Si • Sj 



Si • S' 
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(A3) 
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While the first two terms in (A3) cancel for the magn etic interactions in the limit of 77 ^ 0, the last term favors AF 
spin orieBtation. We recognize that Hamiltonian (A3) describes the superexchange with the superexchange constant 
of 4t^/C/B3'Ea However, for convenience we define the energy unit as J = /U in the present paper. Although the form 
(A3) might in principle be used for further analysis , we prefer to make an expansion of the excitation energies e„ p. 
the denominators for small Jh, and use 77 — Jh/U (2.7) as a pa ram eter which quantifies the Hund's rule exchange.E3 
Using the explicit form of the orbital projection operators Pia ( |A2| ) this results in the following form of the effective 
Hamiltonian for the bond [ij) \\ c. 



{i-j) 



J 



(l + ry) [S.,-S, + 1)~(S., 



(A4) 



which may be further simplified to the form 



+ il[S,- Sj) (rt + r; - 1) 



(A5) 



The first line represents the AF superexchange interactions oc J, while the other two lines describe the weaker FM 
interactons cx J77, and stand for the corrections due to the multiplet splittings of the excited states. 

It is straightforward to verify that the above form of the effective Hamiltonian simplifies in the limit of occupied 
\z) orbitals to 



{if, 



= 4j(l-i 



S,, 



(A6) 



and one recognizes the same constant — i, and the same superexchange interaction 4 J — At^ /U as in the t — J model 
at half-filling .E3 However, the effective superexchange is somewhat reduced by the factor (1 — ^77) in the presence of 
the Hund's rule interaction. 

The effective interactions on t he bonds within the (a, 6) planes may be now obtained by rotating Eq. (A4) the 
orbital operators on Eq. (A4) by 7r/2 to the cubic axes a and h which generates the orbital operators and 
( ^.12 ), r espec tively. This results in a nontrivial coupling between the orbital and spin degrees of freedom, as given 
in Eq. (2.10). We note that in the case of a single s-orbital per site, it would suffice to rotate instead the simpler 



projected form (A6), which would give the same superexchange interaction in any direction. 



APPENDIX B: COMMUTATION RULES IN THE SO(4) ALGEBRA FOR THE SPIN-ORBITAL MODEL 

In order to illustrate the full algebraic structure of our problem, we present here the so(4) commutators between 
the various excitation operators which are needed for calculating the excitation spectra in Sec. As the operators 
defined on different sites commute, we only specify the on-site commutators. 

The spin operators fulfill the usual relations for each orbital a = x, z, 

rc+ 02 1 ^ _ c+ 

Their commutators with the other operators which describe either spin-and-orbital (transverse), or orbital (longitu- 
dinal, i.e., excitonic) excitations are responsible for the coupling between spin- and spin-and-orbital excitations (here 
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['5+„,T.^o] = -K5c., (B2) 
while they commute with the orbital-polarization operator, 

[^ia-M-O. (B3) 

The operators for spin-and-orbital excitations have the following commutators: (i) with the spin operators, 

\Klp^S-pp\^Ktf,^ + T,f,^, (B4) 

(ii) with the spin-and-orbital operators. 





= 0, 








= 0, 








= 0, 







.-Stpf,), (B5) 



and (iii) with the orbital-polarization operator. 



IXZ 

[K+,,n,^]=+K+^. (B6) 



The relevant excitonic operators in the symmetry-broken state (4.11) commute with the above spin-transverse 
operators, and K^^, and give the following commutators with the remaining spin- longitudinal operators, 

[TiaPcT, Si^pp] = +^Ao-TiQ,^(j: 

[Tiapa, K^^p] = 0, 

[T^aPa, K^p^] = \{S+^o, + Stpp) + \\a{n^a " n,p), 

[Tiapa, Tiap] ~ 0, 

[Ttapa, T,pa] = ^^criS^^^ + S+^p) + \{n^ia " n^p), 

where Ao- — ±1 for a — |, |. Therefore, the subset of longitudinal operators {Tiapa} generates the excitations which 
do not couple to the transverse excitations. 



APPENDIX C: GREEN FUNCTION EQUATIONS FOR SPIN AND ORBITAL EXCITATIONS 



Here we present the dynamical matrices obtained for the phases with LRO for the spin-orbital model (2.10). It 
is easy to verify that the presented dynamical matrices have an RPA structure and thus describe symmetric spectra 
with respect to a; = 0. 

Let us start with the G-AF phases with either \x) or \z) orbitals occupied. The spin and spin-and-orbital excitations 
are determined from Eqs. ( 1.12| ) and (4.13) for the AFxx phase, and from Eqs. (4.34) and (4.35)-4pithe AFzz phase. 
After using the translational symmetry and performing the familiar RPA decoupling procedure,! 
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(CI) 



where i and j refere to different sites, one finds a system of linear equations for the excitation energies. A straightfor- 
ward but somewhat lengthy calculation shows that the same matrix with different coefficients describes the elementary 
excitations for both AF phases, 















-Rk 



Rk 



■))a \ 
■))a 
■))b 
■))b/ 



0, 



(C2) 



where ujj: is the frequency in units of J, i.e., luj: = ^^J J ■ The constants, Aq and Tq, and the fc-dependent functions P^-j^ 



and Q^fe depend on the consi dered AF phase and are speci fied in Sec. [V, while i?j = |7+(fc). The solution for the 



functions ((5- 

kxx 



eigenenergies is given by Eq. (4.19). As discussed in Section IV A, the same 4x4 matrix equation written down in Eq. 

etcetera, describing the modes generated by the spin-raising operators, is also valid for the Green 

■ ■))a etc., describing the modes generated by the spin-lowering operators {S~.^, K~,^, S~^^, K~^^}, 
with i €z A and j e B, and all transverse modes are doubly degenerate. 

The orbital (longitudinal) excitations correspond to exciting an electron from one orbital to the other without 
changing the spin direction. If A {B) is an up (down) sublattice in the Neel state, the basis operators which define the 
modes are t-spin (|-spin) orbital excitations, as introduced in Sec. IV. One finds the following eigenvalue problem 
using the RPA, 





Pck 
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'Pc.k 
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^ ^"^kzx^ I 



-Pc.k 





V m. 
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■))a 
■))b 
■))b) 



(C3) 



where again is in units of J, i.e., C,-^ = Q]^/ J , and the quantities Ua and p^^ depend on the considered G-AF phase. 

The classical A-AF ground state is discussed here on the example of the MOffa phase. It consists of four sublattices: 
two sublattices [A and B) due to different orbital order in the (a, b) planes (see Fig. ||), and two other (C and D) due 
to spins which alternate along the c-direction. Therefore, one finds an (8 x 8)-matrix which determines the energies of 
the elementary excitations. If the operators transformed to fc-space are ordered as S\^^^^ ^B/ifj.^ ^A^v^ ^Btj,v^ ^Ciifj.^ 
^Diifi^ ^Cfj,v^ ^Diiiy' '^^^ finds a general structure of the eigenvalue problem. 



A 



-B 



B 



-A 



^k^ 



0, 



(C4) 



where A and B are (4 x 4) symmetric matrices, X is the (4 x 4) identity matrix, and — uj^/ J. Using the averages 
of the diagonal operators in the classical ground state. 



one finds the following elements of matrix A^ 
All 



1 

1 ' 



A 



13 



A. 



33 



-422 
Ax2 

-A2A 
AlA 
A2Z 

: An 

^34 



f 2(2- 



2''V2 



-(1 
+ (1 
-(1 



rj) cos 
sin^ 261)] 7+(fc), 



sin^ 26), 



-i(l - ir/)(l - 2cos26l)2 
i(l - i?7)(l - 2cos26l)^ - 
-i(l - i77)sin26'(2- COS26') - i (3 + ^77) sin 461 - ie^ sin 26*, 
2 cos 26*) sin 26*7+ (fc) + ^[l-{2-r]) cos 261] 7_(fc), 
2 cos 26*) sin 26*7+ (fc) + ^ [I - {2 ~ r]) cos 261] 7_(fc), 
i77)(l-2cos26l) + iry- i(l-F2ry)(l ■ 
2cos46')7+(fc), 



2sin^ 26*) - cos 26*, 



and the following nonzero elements of matrix B, 



(C5) 
(C6) 

(C7) 
(C8) 
(C9) 
(CIO) 

(Cll) 
(C12) 
(C13) 
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Sii = S22= [(l-ir?)(l+cos20)-i77] (1 + cos2%,(fc), 
633 = S44 = sin2 20 74fc), 



(CM) 
(C15) 

Si3 =631 = -S24 = -S42 = - (1 - ir? + cos20)sin20 7,(fc). (C16) 

The longitudinal excitations in the A-AF phases were obtained by solving the respective Green function equations 
for the excitation operators ( 4.58| ). After transforming these equations to fc-space, and taking the following sequence 
of excitation operators, TA^l.v], Tb^j,u], TAvfi], T^Buii], Tc^„i, Tonui, Tc^^i, Toi^ni, one finds an eigenvalue problem of 
the form, 




= 0, 



where 7?., and Q are symmetric matrices. Their nonvanishing elements are defined as follows: 

i(l - lrj)[l - 2 cos 461 + 2cos26'(2 + cos 261)] 
fry cos 261 - 1(1 + 2ri) cos 40 - cos 261, 
i (1 + r;) (1-2 cos 20)7+ (fc), 
i(l + r;)(l-2cos20)7+(fc). 



(C17) 
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V12 


= 7^21 = 


7^l2 


= 7^2l = 


Qii 


= Q22 = 



(C18) 

(C19) 
(C20) 
(C21) 



As in the AF phases, the coupling between the sublattices A and C and between B and D, respectively, is proportional 
to the weak FM component, 77. The mechanism of this coupling is explained in Fig. |^. 



APPENDIX D: NEUTRON INTENSITIES IN TRANSVERSE EXCITATIONS 

In this Appendix we explain the intensities x(^) neutron scattering seen in the presence of orbilal degrees of 
freedom. One can start from the general expression for the cross section for pure magnetic scatteringJZa 



dfldui ki 







X -L / dte-^'^'{S,^{0)S,^{t))e-^'^^^^-^'\ (Dl) 
In J 

where fcg and ki are the initial and final momenta, while q is the momentum transfer. The spin components at site 



i and j are perpendicular to q. By integrating over time t one finds that the neutron cross section (Dl) is related to 
the imaginary part of the spin-spin Green function, 



da ki 



cx 



dVLduj ko 



■iq(Ri-Rj) 



><^2Im|^((5jl|SS.))-.|e(^), (D2) 

where Q{lo) = 1 for w > 0, and Q{Ld) = for w < 0, and we took the limit of temperature T ^ 0. In order to extract 
the perpendicular component of the spin-spin correlation function from the Green functions, ((S'jY |'5'j'j_))-w, we use 
the identity. 

The components of the Green functions in q-space, {{S'i\S^^)-uJ^ are found using the following properties of the 
transverse spin-spin functions. 
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{{S+\S+.))^ = {{S^\SZ^))^ = 0, (D4) 

and = for the wave-vectors q^Q, where Q is the nesting vector. One finds that the neutron cross 

section normalized per one site may be written as follows, 

1 (fa I fci f 



cx 



N d^dui Svr fco N 



^i-NY.r,{q)mx{q), (D5) 



where x(<f) is the neutron scattering intensity which includes the geometrical factor which originates from Eq. ( |D3| ). 
It is proportional to a linear combination of the diagonal and off-diagonal elements of the Green function, and one 
finds for a two-sublattice magnetic structure, as for example in AFxx and AFzz phases, 

X{q) = + ^) 2Im [GAA{q. -OJ) + Gbb{1 -co) 

+GAB{q, + GBA{q, -UJ)] eiio), (D6) 

with the element 6*^^(9, —to) standing for the transverse Green function, {{S^ ^\S^ etcetera, and the indices 

A and B refer to two sublattices. The explicit formula in terms of the spectral intensities Amliiq) is given by 



+ A^^iiq)] Sic - J;^). (D7) 



We have used Eq. (D7) to determine the contributions to the neutron cross-section due to different excitations, as 
analyzed in Sec. IV and presented in Figs. |9|-[l^. The generalization to the case of four-sublattice structures found in 
the MOffa and MOaff phases is straightforward. 
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TABLE I. Individual contributions to quantum corrections {53") of the AF order parameter in AFxx {E^ > 0) and AFzz 
[Ez < 0) phases due to: spin-wave {{S~ S^)), spin-and-orbital-wave {{K~ K'^)), and the leading contribution from low-energy 
mode, {SS^)i. The values of the order parameter in RPA are given by (5'^)rpa- 



Jh/U 


EJJ 


{S-S+} 


{K-K+) 






(S'^)rpa 


0.0 


-3.0 


0.2680 


0.0117 


0.2731 


0.2797 


0.2203 


0.0 


-2.0 


0.2733 


0.0187 


0.2606 


0.2920 


0.2080 


0.0 


-1.0 


0.2839 


0.0368 


0.2146 


0.3207 


0.1793 


0.0 


1.0 


0.2645 


0.0901 


0.2440 


0.3546 


0.1454 


0.0 


2.0 


0.2416 


0.0516 


0.2426 


0.2932 


0.2068 


0.0 


3.0 


0.2298 


0.0352 


0.2455 


0.2650 


0.2350 


0.1 


-3.0 


0.2919 


0.0140 


0.2963 


0.3059 


0.1941 


0.1 


-2.0 


0.2995 


0.0245 


0.2757 


0.3240 


0.1760 


0.1 


-1.0 


0.3188 


0.0612 


0.2339 


0.3800 


0.1200 


0.1 


1.0 


0.2925 


0.1461 


0.2864 


0.4387 


0.0613 


0.1 


2.0 


0.2519 


0.0665 


0.2493 


0.3183 


0.1817 


0.1 


3.0 


0.2352 


0.0421 


0.2519 


0.2773 


0.2227 


0.2 


-3.0 


0.3270 


0.0174 


0.3291 


0.3445 


0.1555 


n 9 
u.z 


9 n 
-z.u 


U.ooaO 


U.UoOi 


U.oUZO 


yj.o 1 ou 




0.2 


2.0 


0.2687 


0.0928 


0.2647 


0.3615 


0.1385 


0.2 


3.0 


0.2428 


0.0521 


0.2593 


0.2950 


0.2050 


0.2 


10.0 


0.2071 


0.0092 


0.2077 


0.2163 


0.2837 


0.3 


-3.0 


0.3861 


0.0232 


0.3834 


0.4093 


0.0907 


0.3 


-2.0 


0.4215 


0.0601 


0.3720 


0.4816 


0.0184 


0.3 


2.0 


0.3026 


0.1530 


0.3179 


0.4556 


0.0444 


0.3 


3.0 


0.2545 


0.0680 


0.2706 


0.3224 


0.1776 


0.3 


10.0 


0.2076 


0.0097 


0.2083 


0.2173 


0.2827 



TABLE II. Individual contributions to the quantum corrections of the magnetic order parameter ((5iS^) in MO phases due 
to spin-wave, {S~S'^), and due to spin-and-orbital-wave excitations, (ICIC^), and due to individual modes as labelled in Figs. 
|ll] and |l^, {SS^)n, respectively. The values of the renormalized order parameter in RPA are given by (<S^)h,pa. 



Jh/U 


E,/J 


(5-5+> 






(55">2 








(<S^)r,pa 


0.2 


0.0 


0.1350 


0.0508 


0.0114 


0.0344 


0.0709 


0.0691 


0.1858 


0.3142 


0.3 


-2.0 


0.2138 


0.0323 


0.0673 


0.0646 


0.0585 


0.0557 


0.2461 


0.2539 


0.3 


-1.0 


0.1338 


0.0336 


0.0411 


0.0025 


0.0547 


0.0691 


0.1674 


0.3326 


0.3 


0.0 


0.0918 


0.0354 


0.0122 


0.0241 


0.0425 


0.0485 


0.1273 


0.3727 


0.3 


1.0 


0.1095 


0.0323 


0.0285 


0.0041 


0.0684 


0.0408 


0.1418 


0.3582 


0.3 


2.0 


0.1330 


0.0328 


0.0327 


0.0076 


0.0754 


0.0502 


0.1658 


0.3342 


0.3 


3.0 


0.1664 


0.0329 


0.0465 


0.0146 


0.0738 


0.0644 


0.1993 


0.3007 


0.4 


-3.0 


0.2144 


0.0232 


0.0876 


0.0958 


0.0294 


0.0249 


0.2376 


0.2624 


0.4 


-2.0 


0.1373 


0.0258 


0.0552 


0.0145 


0.0453 


0.0482 


0.1631 


0.3369 


0.4 


-1.0 


0.0928 


0.0269 


0.0370 


0.0020 


0.0302 


0.0505 


0.1197 


0.3803 


0.4 


0.0 


0.0647 


0.0274 


0.0224 


0.0080 


0.0257 


0.0360 


0.0921 


0.4079 


0.4 


1.0 


0.0776 


0.0254 


0.0258 


0.0038 


0.0494 


0.0240 


0.1030 


0.3970 


0.4 


2.0 


0.0924 


0.0258 


0.0292 


0.0063 


0.0552 


0.0276 


0.1182 


0.3818 


0.4 


3.0 


0.1117 


0.0259 


0.0363 


0.0104 


0.0590 


0.0319 


0.1376 


0.3624 
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TABLE III. The mean- field energy, Emf, the quantum 
energy correction due to transverse modes and due to lon- 
gitudinal modes, 5Et and SEi, respectively, and the ground 
state energy in RPA, -ZJrpa (all in the units of J). The labels 
FFA and AFF indicate the way of staggering of FM planes in 
the MO phases with A-AF order. 



Jh/U 


E,/J 


Emf 


5Et 


5Ei 


-£'RPA 


phase 


0.0 


-2.0 


-4.0000 


0.6440 


0.0 


-4.6440 


AFzz 


0.0 


-1.0 


-3.5000 


0.6700 


0.0 


-4.1700 


AFzz 


0.0 


1.0 


-3.5000 


0.7073 


0.0 


-4.2073 


AFxx 


0.0 


2.0 


-4.0000 


0.6399 


0.0 


-4.6399 


AFxx 


0.1 


-2.0 


-3.9250 


0.6354 


0.0008 


-4.5612 


AFzz 


0.1 


-1.0 


-3.4250 


0.6735 


0.0021 


-4.1006 


AFzz 


0.1 


1.0 


-3.4250 


0.7344 


0.0020 


-4.1614 


AFxx 


0.1 


2.0 


-3.9250 


0.6384 


0.0008 


-4.5642 


AFxx 


0.2 


-3.0 


-4.3500 


0.6082 


0.0024 


-4.9606 


AFzz 


0.2 


-2.0 


-3.8500 


0.6328 


0.0042 


-4.4870 


AFzz 


0.2 


-1.0 


-3.4769 


0.3964 


0.0009 


-3.8742 


FFA 


0.2 


0.0 


-3.2558 


0.2992 


0.0028 


-3.5577 


FFA 


0.2 


1.0 


-3.3543 


0.3437 


0.0010 


-3.6990 


AFF 


0.2 


2.0 


-3.4769 


0.3962 


0.0005 


-3.8738 


AFF 


0.2 


2.0 


-3 8500 


0.6472 


0.0041 


-4.5013 


AFxx 


0.3 


-3.0 


-4.2750 


0.6052 


0.0062 


-4.8864 


AFzz 


0.3 


-3.0 


-4.2272 


0.5252 


0.0194 


-4.7717 


FFA 


0.3 


-2.0 


-3.7750 


0.6419 


0.0134 


-4.4303 


AFzz 


0.3 


-2.0 


-3.8651 


0.3944 


0.0037 


-4.2632 


FFA 


0.3 


-1.0 


-3.5892 


0.3040 


0.0019 


-3.8951 


FFA 


0.3 


0.0 


-3.3996 


0.2335 


0.0054 


-3.6384 


FFA 


0.3 


1.0 


-3.4836 


0.2664 


0.0031 


-3.7531 


AFF 


0.3 


2.0 


-3.5892 


0.3038 


0.0016 


-3.8947 


AFF 


0.3 


2.0 


-3.7750 


0.6768 


0.0134 


-4.4652 


AFxx 


0.3 


3.0 


-3.7164 


0.3459 


0.0015 


-4.0638 


AFF 


0.3 


3.0 


-4.2750 


0.5773 


0.0063 


-4.8586 


AFxx 


0.3 


10.0 


-7.7750 


0.4048 


0.0014 


-8.1812 


AFxx 
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FIG. 1. Virtual transitions d^d^ — > df^d^ which lead to 
a spin-flip and generate effective interactions for a bond 
(ij) II c-axis, with the excitation energies at Ez = 0. For 
two holes in different orbitals (a), either the triplet "^Az or 
the interorbital sing let ^Ee occurs as an intermediate d con- 
figuration, while if both holes are in \z) orbitals (b), two other 
singlets, ^E^ and ^Ai, with double occupancy of \z) orbital, 
contribute. The latter processes are possible either from i to 
j or from j to i. 



FIG. 2. Energies of the virtual excitations Ei/U shown in 
Fig. |l| as functions of ^E^/J for Jh/U = 0.3. The lowest 
triplet \'^A2) state is indicated by full circles, and the singlet 
states (I^S) and I^Ai)) by fuU lines. 

FIG. 3. Schematic representation of orbital and magnetic 
long-range order within the (a, b) planes of AFxx, AFzz, 
MOffa, and MOaff phases, respectively. The shadded parts 
of different orbitals are oriented along the c-axis. The spins 
(arrows) in the next (a, 6) plane in the c-direction are AF to 
those below them in AFzz and MOffa phases, and FM in 
MOaff phase. In the AFxx phase there is no magnetic cou- 
pling to the next plane along the c-axis, but this degeneracy 
is removed in MOaaf phase, where a small \z) component 
promotes a FM coupling. 



FIG. 8. Schematic propagation of the orbital (excitonic) 
excitation (a). If Jh = 0, an orbital excitation can propa- 
gate only to state (b) and is accompanied by a spin-flip (top) , 
while Jh > allows also the spin-flip in the intermediate df 
state, and thus the propagation without spin-flip (c) becomes 
possible (bottom). 

FIG. 9. Lower panel: spin-wave and spin-and-orbital-wave 
transverse excitations (full lines) and longitudinal excitations 
(dashed lines) in AFxx phase; upper panel: neutron intensi- 
ties of the transverse excitations. Parameters: Ez/J = 3.0 
and Jh/U = 0.3. 

FIG. 10. The same as in Fig. ^ but for the AFzz phase, 
as obtained for Ez/J — —3.0 and Jh/U = 0.3. 

FIG. 11. The same as in Fig. ^, but for the MOffa phase, 
as obtained for Ez/J — —1.0 and Jh/U = 0.3. Different 
modes are labelled by the increasing indices i = 1, . . . , 4 with 
increasing energy. 

FIG. 12. The same as in Fig. |, but for the MOAFF phase, 
as obtained for Ez/J — 1.0 and Jh/U — 0.3. 



FIG. 4. Mean-fleld phase diagram of the 3D spin-orbital 
model (2.9) in the {Ez,Jh) plane (/3 = 1). The lines sepa- 
rate the classical states shown in Fig. the transition from 
AFxx to MOaff phase is second order (dashed line), while 
the remaining transitions are flrst order (full lines). 



FIG. 13. Spin-wave and spin-and-orbital-wave excitations 
in the G-AF phases: AFxx (left) and AFzz (right), in the 
main directions of the 3D BZ for a few values of Ez (in the 
units of J), and for Jh/U — 0.3. The lower-energy mode 
becomes soft for Ez/J < 1.54 {Ez/J > -1.84) in the AFxx 
(AFzz) phase. 



FIG. 5. Mean-field phase diagrams of the spin-orbital 



model (2.9) in the {Ez, Jh) plane for different values of hop- 
ping along c-axis: (a) /3 = 1.414, and (b) /3 = 0.707. The 
magnetic phases and lines as in Fig. 0. 



FIG. 14. The same as in Fig. |l^, but without the coupling 
between the spin-wave and spin-and-orbital-wave excitations 
in both G-AF phases: AFxx (left) and AFzz (right). 



FIG. 6. Mean-fleld phase diagram of the spin-orbital model 
( ^.S| ) in the {Ez,Jh) plane in two dimensions (/3 — 0). Full 
lines separate the classical states AFxx, AFzz, and MOff 
shown in Fig. ^, while the spin order in the MOaa phase is 
AF, and the orbitals are in between those in AFxx and MOff 
phase. 

FIG. 7. Schematic representation of the mixed orbitals in 
(a, b) planes of the MOff phase in a 2D model: (a) the or- 
bitals with their phases, and (b) the resulting distortion in 
the oxygen lattice, stabilized by the orbital ordering. 



FIG. 15. Transverse (full lines) and longitudinal (dashed 
lines) excitations in MOffa phase in the main directions of 
the 3D BZ for a few values of Jh/U, and for Ez/J = —0.5. 
The lower-energy mode becomes soft for Jh/U < 0.06 . 



FIG. 16. Renormalization of the magnetic LRO parame- 
ter (Sf ) by quantum fluctuations as obtained in RPA in: (a) 
AFzz (left) and AFxx (right) phases as functions of Ez/J for 
Jh/U = 0.1 and 0.3; (b) MOffa phase as function of Jh/U 
for Ez/J = 0.5, -0.5 and -1.5. 

FIG. 17. Renormalization of the magnetic LRO param- 
eter {Si) by quantum fluctuations obtained for the G-AF 
phases as in Fig. |l6|(a), but for decoupled spin- wave and 
spin-and-orbital-wave excitations shown in Fig. [14l 
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FIG. 18. Average density of |2;)-holes {rix) as obtained for 
Jh/U = 0.3 in MF approximation (dashed lines) and with 
the quantum corrections calculated in RPA (full lines). The 
splitting of lines for E^/ J > corresponds to the MOaff 
phase with two different hole densities {nx)A 7^ ("-i^)s on the 
ions belonging to two sublattices (see Fig. 0). 



FIG. 19. Schematic phase diagram of the spin-orbital 
model including quantum fluctuations. The spin liquid phase 
is expected to separate the AF phases with different types 
of magnetic LRO: G-AF phases with either d^i_y2 {\xx)) or 
1^322 {\zz)) orbital occupied on both sublattices from the 
A-AF phases with mixed orbitals (MO) ordered on two sub- 
lattices. 
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